Commit Graph

378 Commits

Author SHA1 Message Date
Jun Wu
5c77fa2aed dag: implement RemoteIdConvertProtocol on dag snapshots
Summary: This will be used in upcoming changes. It just delegates to the Arc inner.

Reviewed By: sfilipco

Differential Revision: D27629313

fbshipit-source-id: ba6cd7cac2b9f5c1a2898c439c53768995a2dc42
2021-04-13 16:37:47 -07:00
Jun Wu
f4bc4b88b9 dag: add DagAlgorithm::is_vertex_lazy
Summary: This will be used by upcoming changes.

Reviewed By: sfilipco

Differential Revision: D27629312

fbshipit-source-id: 6c56e73caf4e1a398ac3a8b4294bd9f380af3764
2021-04-13 16:37:47 -07:00
Jun Wu
46fa8482a1 dag: add RemoteIdConvrtProtocol::is_local
Summary: This will be used by upcoming changes.

Reviewed By: sfilipco

Differential Revision: D27629319

fbshipit-source-id: d19e490268561f3154642e5bb1e415da4c5d03ee
2021-04-13 16:37:47 -07:00
Jun Wu
db51ea8228 dag: do not resolve id or names remotely inside a nonblocking context
Summary:
Otherwise it might panic (ex. calling into tokio without entering a tokio
runtime). This can happen in, for example, `Debug::fmt(&IdStaticSet, ...)`.

Reviewed By: sfilipco

Differential Revision: D27581487

fbshipit-source-id: feec53e088706adcc6710afcf24fa70598f886cf
2021-04-13 16:37:46 -07:00
Jun Wu
1038af19fa dag: add "dirty" to DagAlgorithm
Summary:
This will be used by "add_heads" logic to detect what vertexes to insert
and might reduce remote fetches.

Reviewed By: sfilipco

Differential Revision: D27572359

fbshipit-source-id: d0bf027a69d180663a1587dfde613cb76b05072a
2021-04-13 16:37:46 -07:00
Jun Wu
1913d17c4e dag: add "dirty" to IdDagAlgorithm
Summary: The API returns entries buffered in memory not persisted.

Reviewed By: sfilipco

Differential Revision: D27572360

fbshipit-source-id: 555988f7c891f2d928bfa1e486a0fc1d089b4ad5
2021-04-13 16:37:46 -07:00
Jun Wu
ce8bfb19a4 dag: add ways to get the minimal "dirty" Id in IdDag
Summary: This will be used to select "dirty" (not written to disk) set in the IdDag.

Reviewed By: sfilipco

Differential Revision: D27572361

fbshipit-source-id: 0b4d2e092ece835e3d4b6aa831d32ffffc7087ca
2021-04-13 16:37:45 -07:00
Jun Wu
9974efcfed dag: consider overlap IdMap for prefix lookup
Summary:
Before this change, overlap IdMap was not considered for prefix lookup. That
results in "shortest" template not working and smartlog prints full hashes
for remote/stable etc.

Reviewed By: sfilipco

Differential Revision: D27547582

fbshipit-source-id: 7a56590775eed9d509f2212f8e5009c04aaf4e9d
2021-04-13 16:37:45 -07:00
Jun Wu
d450aa7bd4 dag: extract prefix lookup logic from MemIdMap to CoreMemIdMap
Summary: It will be reused in NameDag.

Reviewed By: sfilipco

Differential Revision: D27547583

fbshipit-source-id: da85fc7504d20877210e8ed1a97cbec18d06eede
2021-04-13 16:37:45 -07:00
Jun Wu
2ab2ba667c dag: avoid SyncNameSetQuery in namedag and ops
Summary:
Now NameSet iteration can be blocking, SyncNameSetQuery is no longer sound.
Remove SyncNameSetQuery in key logic (namedag and ops) and replace them with
async logic.

Reviewed By: sfilipco

Differential Revision: D27547581

fbshipit-source-id: af69b1a8219e97d10278939407ee79f9b333a77f
2021-04-13 16:37:45 -07:00
Jun Wu
af3c3b3fd0 dag: trigger remote fetching in various dag algorithms
Summary: Dag algorithms like `parent_names` need to fetch vertexes via remote automatically.

Reviewed By: sfilipco

Differential Revision: D27547584

fbshipit-source-id: 8106931d6f642c9a4bf0f3c546ba881c2ca73fea
2021-04-13 16:37:45 -07:00
Jun Wu
7508e2f81f dag: fix filter set's contains implementation
Summary:
The "filter" set's filter function might not be prepared for inputs outside
the parent set. So let's the "contains" function to test against the parent
set first, then test the filter function.

This fixes the "merge()" set's "contains" check using the revlog backend:

  In [1]: v=repo.revs('draft() & merge()')

  In [2]: v._set
  Out[2]: <meta ?>

  In [3]: m.node.nullid in v._set
  Out[3]: False

Before this diff it would be:

  In [3]: m.node.nullid in v._set
  CommitLookupError: '0000000000000000000000000000000000000000 cannot be found'

Note: Segmented changelog backend is not affected because it does not use
filter sets.

Reviewed By: xavierd

Differential Revision: D27657502

fbshipit-source-id: 30bb261fea59bdc5644580e98796f52fa93c2705
2021-04-08 12:25:51 -07:00
Jun Wu
6cba98986a dag: fix path response with batch_size >= 2
Summary:
The issue is that `mut i: usize` is no longer shared across multiple `async
move` blocks (introduced by D27308798 (0df4efa969)).

Rewrite the logic to collect `ids` first, then use `vertex_name_batch`
query instead.

Reviewed By: sfilipco

Differential Revision: D27406586

fbshipit-source-id: b41fe3a13114dc34aa5763e6e2bebe0571decc87
2021-04-05 12:55:41 -07:00
Jun Wu
9342556e54 dag: merge adjacent x~n queries
Summary:
Merge paths like `x~n` and `x~(n+1)` to `x~n (batch_size = 2)`.
This could be more efficient bandwidth-wise and algorithm-wise.

Reviewed By: sfilipco

Differential Revision: D27406587

fbshipit-source-id: f2a67352ad627945685e33667e8299a2bc652930
2021-04-05 12:55:40 -07:00
Jun Wu
775899c0f2 dag: make protocol use IdSet instead of Vec<Id>
Summary: IdSet is more compact. This changes the order a bit.

Reviewed By: sfilipco

Differential Revision: D27339279

fbshipit-source-id: e9b50a47beba081b892eccd7711dbd6ab5c3a886
2021-04-05 12:55:40 -07:00
Jun Wu
c515d1f54f dag: show AnestorPath batch size in debug output
Summary: This will be used by the next change.

Reviewed By: sfilipco

Differential Revision: D27406591

fbshipit-source-id: fcacc35a9ae8ed96cebb2af804d26d1e5e83ad9e
2021-04-05 12:55:40 -07:00
Jun Wu
95ece1d6fe dag: add a way to flush the overlay map
Summary:
Add a way to flush the overlay map to disk so we can avoid network fetches over
and over.

Reviewed By: sfilipco

Differential Revision: D27406592

fbshipit-source-id: 7086ad665119cc3a0834f533690325c7a2363442
2021-04-05 12:55:40 -07:00
Jun Wu
dd042424f3 dag: move (x~n, name) -> (id, name) calculation to a function
Summary: It will be reused elsewhere.

Reviewed By: sfilipco

Differential Revision: D27406593

fbshipit-source-id: 296cf5f50830bb7285e0cb9c7c15a9b374689819
2021-04-05 12:55:40 -07:00
Jun Wu
5326b18c2b dag: track x~n paths in NameDag
Summary:
I spent some time thinking about how to flush the "overlay_map" to disk.
It is a bit tricky because the on-disk IdMap might have changed in an
incompatible way. I tried a few ways to verify the on-disk IdMap remains
compatible and didn't find a way that looks good (either slow - calculating
universal_ids, or is not 100% correct in corner cases).

Now I come up with this "just track x~n" idea. It trades memory usage (~2x
overlay_map) for easy-to-verify correctness, and efficient overlay_map
flush.

Reviewed By: sfilipco

Differential Revision: D27406583

fbshipit-source-id: 0b7fb3186a9c15f376c1dc4afe7f0516c25d3dec
2021-04-05 12:55:39 -07:00
Jun Wu
e6d231818d dag: add more comments about NameDag locking
Summary: It is not obvious. So let's add more comments.

Reviewed By: sfilipco

Differential Revision: D27406584

fbshipit-source-id: 9ce1215efc1a6d4849180c6693616613c08f2a51
2021-04-05 12:55:39 -07:00
Jun Wu
2b5f78d0ac dag: add a test about sparse dag
Summary:
A sparse dag does not have full IdMap. Its IdMap only contains "universally known" entries.

Add a basic test about cloning from a sparse clone data and resolve vertex <-> id mapping
on the fly.

Reviewed By: sfilipco

Differential Revision: D27352018

fbshipit-source-id: 4a3f5f50be52e91bf7b2021cdc858bcab9c99e80
2021-04-05 12:55:39 -07:00
Jun Wu
d5b5e1ea93 dag: make import_clone_data flush the dag directly
Summary:
The `NameDag::flush` API will actually rebuild the graph using a "parent" function.
That is not necessary if we got clone data, and won't work well for a lazy graph
(since the parent function talks about vertex names and some names are missing).

Let's bypass the `flush` function and write data directly in `import_clone_data`.

Reviewed By: sfilipco

Differential Revision: D27352019

fbshipit-source-id: a79569d25d858447b8c5eb86902b8d39ae0429a3
2021-04-05 12:55:39 -07:00
Jun Wu
cdbc0b9bb1 dag: add ways to use a NameDag as an implementation of the remote protocols
Summary: This will be used in tests.

Reviewed By: sfilipco

Differential Revision: D27343882

fbshipit-source-id: 5a2d94a9f755eed0fc27e5a11093b55c810dc8da
2021-04-05 12:55:39 -07:00
Jun Wu
13c9880eca dag: add logic to export clone data
Summary:
Implement logic to export the clone data. This allows us to construct a sparse/lazy
dag via export + import CloneData.

Reviewed By: sfilipco

Differential Revision: D27343885

fbshipit-source-id: 71dc0d31e36876a8b6a8c3d7f3498be3262ce297
2021-04-05 12:55:39 -07:00
Jun Wu
ec2c1a7928 dag: add verification importing clone data
Summary:
Clone data can only be imported to an empty Dag and universally known vertexes
should be present in the IdMap.

Reviewed By: sfilipco

Differential Revision: D27343888

fbshipit-source-id: ba150d6afdbe15f0902ec20ff150a70657e24c80
2021-04-05 12:55:39 -07:00
Jun Wu
429b0e1e15 dag: make import_clone_data async
Summary: It'll use some async functions.

Reviewed By: sfilipco

Differential Revision: D27406585

fbshipit-source-id: e757796f712a5f95f1227f88e797e43551039f0b
2021-04-05 12:55:38 -07:00
Jun Wu
fed4cdbe50 dag: implement Id prefetch for IdStatic set
Summary: Make IdStatic prefetch Id -> Names on iteration.

Reviewed By: sfilipco

Differential Revision: D27343886

fbshipit-source-id: 7957b574c8c14cfea476b9c42cbf9f11fefa39be
2021-04-05 12:55:38 -07:00
Jun Wu
8364f23186 dag: make IdConvert batch API on NameDag use less network requests
Summary: Collect "missing" items and only use one request to fetch them.

Reviewed By: sfilipco

Differential Revision: D27406588

fbshipit-source-id: a5cd091b39d90c1ad0e7c5d509673c4665232304
2021-04-05 12:55:38 -07:00
Jun Wu
55b0132518 dag: implement ExactSizeIterator on SpanSet
Summary: This allows something like `iter.rev().take(n).rev()`.

Reviewed By: sfilipco

Differential Revision: D27343887

fbshipit-source-id: 06c095eb448272dca6add0e707cdf38f0daee252
2021-04-05 12:55:38 -07:00
Jun Wu
0326d38c75 dag: add batch id <-> name API in IdConvert
Summary:
This will be used by upcoming changes. Sparse/Lazy NameSet will override it
to reduce network round-trips.

Reviewed By: sfilipco

Differential Revision: D27406590

fbshipit-source-id: a44a73b4aec6e14d6e82d55285fe1cfc0fcfd482
2021-04-05 12:51:38 -07:00
Jun Wu
7c8056ae38 dag: add APIs to test if Id or Vertex is present in IdMap locally
Summary: This will be used by upcoming changes.

Reviewed By: sfilipco

Differential Revision: D27343884

fbshipit-source-id: 0938b1fb3d90b35f9d51c468cffca53e3f421bb8
2021-04-05 12:51:38 -07:00
Jun Wu
0ff3c9109d dag: make NameDag::IdConvert resolve vertexes remotely
Summary:
Remove some TODOs. This serves as fallback paths where batch prefetch didn't happen.
I'd expect most use-cases will trigger IdStatic set's batch prefetch logic (to be
added).

I haven't decided what to do exactly with "contains". Fetching remotely there seems
to require some kind of negative cache (ex. in mutation records there might be nodes
not in the graph). But it _might_ be okay to say the "contains" is a local-only
operation, too. I leave it as-is and we'll see how the Python world uses "contains"
later.

Reviewed By: sfilipco

Differential Revision: D27339275

fbshipit-source-id: ba70b3c84a391a8e395c73ccd1d7e08f92b0cbd0
2021-04-05 12:51:37 -07:00
Jun Wu
b7c63d192d dag: add methods to resolve id <-> names remotely on NameDag
Summary:
Put everything together. I used "programming error" extensively to
provide more context if we have to investigate issues in the future.

Reviewed By: sfilipco

Differential Revision: D27339278

fbshipit-source-id: 574a2c048dc1d24dbe690f862fec3e5078cb067a
2021-04-05 12:51:37 -07:00
Jun Wu
8a381893db dag: improve error message in protocol
Summary: Provide more context about what invariants we expect. Not just show "vertex not found".

Reviewed By: sfilipco

Differential Revision: D27339273

fbshipit-source-id: 1c6c92537ff37666ff603783adfd8f9ea770fbaa
2021-04-05 12:51:37 -07:00
Jun Wu
c85c750baa dag: add a remote protocol field to NameDag
Summary:
Makes NameDag own the state (logic) about how to send remote requests.
So NameDag can send requests on its own.

Reviewed By: sfilipco

Differential Revision: D27339282

fbshipit-source-id: 3cb6327dfeaefae45d4e7b88a3535463a84b195b
2021-04-05 12:48:34 -07:00
Jun Wu
79b40c5ce8 dag: define remote protocol
Summary: Define a trait for implementing the remote protocol elsewhere.

Reviewed By: sfilipco

Differential Revision: D27339281

fbshipit-source-id: da5b316d98863507361d3bde4988fd6c9098f48c
2021-04-05 12:48:34 -07:00
Jun Wu
31ab817ba6 dag: make NameDag IdConvert consider overlay IdMap
Summary: This will make the overlay IdMap effective when query against the NameDag.

Reviewed By: sfilipco

Differential Revision: D27339276

fbshipit-source-id: 80712bf651beb6c7e9f23bd4233c6d916101696a
2021-04-05 12:48:34 -07:00
Gus Wynn
fc46c24e8f update tokio to 1.4.0
Summary:
https://github.com/tokio-rs/tokio/releases/tag/tokio-1.4.0
I want the `biased;` option in `tokio::select!`

Reviewed By: ahornby

Differential Revision: D27435341

fbshipit-source-id: c29ca954c319327f62466131ae04483ad091bf49
2021-03-31 10:44:20 -07:00
Jun Wu
a008135ba2 dag: expand IdConvert delegation on NameDag
Summary: Expand the IdConvert implementation so we can change it.

Reviewed By: sfilipco

Differential Revision: D27339280

fbshipit-source-id: eb55c63529c895502a25bb279bcba3c13737452a
2021-03-30 09:58:29 -07:00
Jun Wu
a77579877b dag: add overlay IdMap to NameDag
Summary:
Add an overlay IdMap field for NameDag to store temporary remote IdMap results.
This diff just adds the field. It's not used yet.

Reviewed By: sfilipco

Differential Revision: D27339274

fbshipit-source-id: dbbde227f26de15d10c84f5d7c61ca8054577752
2021-03-30 09:58:29 -07:00
Jun Wu
c8b78b3c03 dag: extract in-memory IdMap to a separate structure
Summary:
In a future diff, AbstractNameDag wants an "overlay" IdMap to store temporary
remote IdMap results. The MemIdMap is suitable but has extra features (next
free id, version). Extract the core in-memory IdMap logic for the "overlay"
purpose.

Reviewed By: sfilipco

Differential Revision: D27339277

fbshipit-source-id: 4e73032b8bc6670264e3fa1dd5515ea3bc853d10
2021-03-30 09:58:29 -07:00
Jun Wu
0df4efa969 dag: asyncize Process::process
Summary:
The `Process::process` contains logic to resolve Id <-> Vertex using remote
service. The remote service is async so let's make `Process` async.

Reviewed By: sfilipco

Differential Revision: D27308798

fbshipit-source-id: 30c2c3eda124d542d0867d278ce56a7a174f33e0
2021-03-30 09:58:29 -07:00
Jun Wu
a904d5606d dag: move MultiLog OpenOptions to a DefaultOpenOptions
Summary:
By implementing DefaultOpenOptions, indexedlog provides `repair()` for free.

Re-export the `Repair` trait so other crates can use `repair()` without
importing indexedlog.

Reviewed By: sfilipco

Differential Revision: D27053352

fbshipit-source-id: 8fa952f0e51e007b9d348bc12699ef1d65000c6b
2021-03-15 18:45:07 -07:00
Jun Wu
ffb23d32a3 dag: make Debug<Segment>::fmt print invalid segments
Summary: This makes it easier to spot problematic segments.

Differential Revision: D27025039

fbshipit-source-id: 44ae9eaf03ca54751b64c0cf977b78737eaa58f0
2021-03-12 17:23:48 -08:00
Jun Wu
ea61594e04 dag: add some validations about segment parents
Summary:
We saw the `unwrap()` panic at `iddag.rs:453`:

  let (new_idx, low, high, parent_count, has_root) = candidate.unwrap();

Reading the code, it seems it can only happen if one of `parents` is `high` in a segment.
Parents in a segment should all be < low.  Add some validations for that.

Reviewed By: sfilipco

Differential Revision: D27021394

fbshipit-source-id: 2c51713dfb8f07f2921229d8866f7e8a0bd9490f
2021-03-12 13:02:50 -08:00
Thomas Orozco
f5f78c4906 third-party/rust: update tokio & tokio-stream
Summary:
Pulling this for those 2 PRs:

- https://github.com/tokio-rs/tokio/pull/3547
- https://github.com/tokio-rs/tokio/pull/3576

Reviewed By: ahornby

Differential Revision: D26944216

fbshipit-source-id: ad67afa69cb291cfb1622cf4b2a10727a13d19cd
2021-03-10 11:01:28 -08:00
Andrey Chursin
0be8e8ce29 vfs: introduce AsyncVfs
Summary:
AsyncVfs provides async vfs interface.
It will be used in the native checkout instead of current use case that spawns blocking tokio tasks for VFS action

Reviewed By: quark-zju

Differential Revision: D26801250

fbshipit-source-id: bb26c4fc8acac82f4b55bb3f2f3964a6d0b64014
2021-03-05 21:47:51 -08:00
Lukas Piatkowski
ad106958f2 eden/scm/lib: autogenerate all Cargo.toml files with autocargo
Summary: This diff removes the split between manually managed and autocargo managed Cargo.toml files in `eden/scm/lib`, now all files are autogenerated.

Reviewed By: quark-zju

Differential Revision: D26830884

fbshipit-source-id: 3a5d8409a61347c7650cc7d8192fa426c03733dc
2021-03-05 04:29:49 -08:00
Jun Wu
30ad76433e dag: add a way to describe bytes in indexedlog IdDag store
Summary:
This will be useful to explain bytes in the indexedlog.

Together with `debugdumpindexedlog` this can be used to troubleshoot issues.
For example, `debugdumpindexedlog` has:

  # Entry 11763:
  0004064b: 00 03 01 00 00 00 00 00 05 f1 00 01 ea 8b 80 80  ................
  0004065b: 80 80 80 80 01                                   .....

The entry can then be easily decoded via debugshell:

  In [1]: def e(s):
   ...:     print(b.dag.describebytes(bin(s.replace(' ',''))))

  In [2]: e('00 03 01 00 00 00 00 00 05 f1 00 01 ea 8b 80 80 80 80 80 80 01')
  # 00: Flags = (empty)
  # 03: Level = 3
  # 01 00 00 00 00 00 05 f1: High = N1521
  # 00: Delta = 0 (Low = N1521)
  # 01: Parent count = 1
  # ea 8b 80 80 80 80 80 80 01: Parents[0] = N1514

Reviewed By: sfilipco

Differential Revision: D26654639

fbshipit-source-id: c8438623b7e22e6abaf5c3011be25587f9d68753
2021-02-25 08:32:48 -08:00
Jun Wu
7684e79be6 dag: fix segment format comment
Summary: The first byte is the flag.

Reviewed By: sfilipco

Differential Revision: D26654640

fbshipit-source-id: 4d66f14c73fe40a154ca7c08f9a9dff3a54ae337
2021-02-25 08:32:47 -08:00