Summary:
Update DagAlgorithm and all its users to async. This makes it easier to make
IdConvert async.
Reviewed By: sfilipco
Differential Revision: D25345236
fbshipit-source-id: d6cf76723356bd0eb81822843b2e581de1e3290a
Summary:
Make it possible to use async functions in MetaSet functions.
It will be used when DagAlgorithm becomes async.
Reviewed By: sfilipco
Differential Revision: D25345229
fbshipit-source-id: 0469d572b56df21fbdbdfae4178377e572adbcda
Summary: This makes it easier to make DagAlgorithm async.
Reviewed By: sfilipco
Differential Revision: D25345234
fbshipit-source-id: 5ca4bac38f5aac4c6611146a87f423a244f1f5a2
Summary: `impl Trait` does not work with `async_trait`.
Reviewed By: sfilipco
Differential Revision: D25345238
fbshipit-source-id: e7890dbaeb162d44e072ea4428d045004608719b
Summary: This makes it easier to migrate to async.
Reviewed By: sfilipco
Differential Revision: D25345228
fbshipit-source-id: e819f0de5f805377a6977325216ef11b14d68c1d
Summary:
Marking IdConvert Sync makes it possible to be used as a trait object with async-trait.
See https://docs.rs/async-trait/0.1.41/async_trait/#dyn-traits
`dag` uses a lot `dyn DagAlgorithm`. In the future when async is used more, the
trait object will be required to be Send or Sync. Just require it on the trait
to make our life easier.
Marking `IdDagStore` as Send + Sync makes async migration easier.
Reviewed By: sfilipco
Differential Revision: D25345231
fbshipit-source-id: 45b96057907cbe2a1d38fd424e7d4c963dd1b245
Summary: Use async function for the PrefixLookup trait.
Reviewed By: sfilipco
Differential Revision: D24840820
fbshipit-source-id: d22cac9f11b06e3127fa956e3f116cf232214125
Summary: This makes the trait objects slightly easier to use.
Reviewed By: sfilipco
Differential Revision: D24840821
fbshipit-source-id: 22fcdf13b62420302b562c309874e08360d02372
Summary: This makes `dyn IdConvert` include `PrefixLookup`.
Reviewed By: sfilipco
Differential Revision: D24840819
fbshipit-source-id: 8d4e25c534f6e4397ec6f643eb3aa116bff12a2c
Summary:
In the future, when async APIs are used, Python bindings will have lifetime
issues. Make it possible to clone the IdMap so the Python bindings can be made
to work.
Reviewed By: sfilipco
Differential Revision: D24840822
fbshipit-source-id: 6aa4e369c877c428ed39d2cbea79e6943836afa8
Summary: This makes NameSet more friendly for async use-cases, interface-wise.
Reviewed By: sfilipco
Differential Revision: D24806695
fbshipit-source-id: 6e640ba2666872a9128d6460e8b53d6a0e595e56
Summary:
Change the main API of NameSet to async. Use the `nonblocking` crate to bridge
the sync and async world for compatibility. Future changes will migrate
Iterator to async Stream.
Reviewed By: sfilipco
Differential Revision: D24806696
fbshipit-source-id: f72571407a5747a4eabe096dada288656c9d426e
Summary:
This method reconstructs a dag from clone data.
At the moment we only have a clone data construction method in Mononoke. It's
the Dags job to construct and import the clone_data. We'll consolidate that at
a later time.
Reviewed By: quark-zju
Differential Revision: D24954823
fbshipit-source-id: fe92179ec80f71234fc8f1cf7709f5104aabb4fb
Summary:
This function is useful in the mononoke to compute the universal commit idmap
that is required for clone.
Reviewed By: quark-zju
Differential Revision: D24808327
fbshipit-source-id: 0cccd59bd7982dd0bc024d5fc85fb5aa5eafb831
Summary:
`flat_segments` are going to be used to generate CloneData. These segments will
be sent to a client repository and are going to bootstrap the iddag.
Reviewed By: quark-zju
Differential Revision: D24808331
fbshipit-source-id: 00bf9723a43bb159cd98304c2c4c6583988d75aa
Summary: This is the object that will be used to bootstrap a Dag after a clone.
Reviewed By: quark-zju
Differential Revision: D24808328
fbshipit-source-id: 2c7e97c027c84a11e8716f2e288500474990169b
Summary:
The goal is to reused the functionality provided by AssignHeadOutcome for clone
purposes.
Reviewed By: quark-zju
Differential Revision: D24717924
fbshipit-source-id: e88f21ee0d8210e805e9d6896bc8992009bd7975
Summary:
I initially saw the incremental build as something that would be run in places
that had IdMap and IdDag stored side by side in process. I am reconsidering
to use incremental build in the tailing process to keeps Segmented Changelog
artifacts up to date.
Since we update the IdMap before we update the IdDag, it is likely that we
will have runs that only update the IdMap and fail to update IdDags. This diff
adds a mechanism for the IdDag to catch up.
Reviewed By: krallin
Differential Revision: D24516440
fbshipit-source-id: 3a99248451d806ae20a0ba96199a34a8a35edaa4
Summary:
I am wondering whether we should customize the serialization format for the
InProcessStore. I want to have a basis for the comparison before I proceed.
Reviewed By: quark-zju
Differential Revision: D24580273
fbshipit-source-id: d3ddfdc029dbdd84f60acace06fddc80b4d005f4
Summary:
This will be used to avoid 1-by-1 fetching for the changelog backend with
commit text stored remotely.
Reviewed By: sfilipco
Differential Revision: D24321293
fbshipit-source-id: 9695c72166cadc0b167e2ce7fde822cdf6b1cea8
Summary:
Turn on rust changelog (changelog2) for all hosts (except hgsql).
Turn on doublewrite backend for hg-dev hosts, triggered by pull.
Tests are mostly working, and I have been using it for weeks.
Reviewed By: singhsrb
Differential Revision: D24259759
fbshipit-source-id: b89a27f98a6d3d1e4ea187bf7b29f875d0e96e2e
Summary: It's no longer useful as the new abstract interface does not need it.
Reviewed By: sfilipco
Differential Revision: D24399516
fbshipit-source-id: 2b6735d2a26706c6a3e6b592d2f3ecfc874c94cb
Summary:
This verifies the abstraction and simplifies the code.
The new code will use non-master segments for add_heads. Therefore the test
changes.
Reviewed By: sfilipco
Differential Revision: D24399496
fbshipit-source-id: 39067ad88ade79b4f7758bcdaafc03e5f34ced91
Summary: This makes the main namedag.rs cleaner. The next step is to move MemNameDag.
Reviewed By: sfilipco
Differential Revision: D24399495
fbshipit-source-id: c1e79a60edd8597fe7264f04548e5312414241a7
Summary: This is the last non-abstract interface of NameDag.
Reviewed By: sfilipco
Differential Revision: D24399514
fbshipit-source-id: f39bb84a1851a4fe4d1f29e6b0961e6a153c943d
Summary:
There is a need to open AbstractNameDag cleanly from a path.
Abstract that.
Reviewed By: sfilipco
Differential Revision: D24399498
fbshipit-source-id: ca242cd929e8f5580120c01eeaa928f630c21ed7
Summary:
I copied the code since it's hard to implement using the macros.
In the future I plan to merge MemNameDag into AbstractNameDag
and remove the macros.
Reviewed By: sfilipco
Differential Revision: D24399517
fbshipit-source-id: 326e76cd06a6e1ad26b39bcb51ba0ff24106c984
Summary: The `delegate!` is updated to support complex `impl`s.
Reviewed By: sfilipco
Differential Revision: D24399518
fbshipit-source-id: b9ba31174472cce4248e9644611cfc207abc3c1d
Summary: Will be used as bounds for abstraction.
Reviewed By: sfilipco
Differential Revision: D24399497
fbshipit-source-id: 343be12237d4850fbde9ebbe4034469527bd77fc
Summary: The `snapshot` field can be used instead.
Reviewed By: sfilipco
Differential Revision: D24399507
fbshipit-source-id: 67de20d897b8b763f724f3ccbd46618dec7911b9
Summary:
The trait requires an `IdMap` snapshot to be locally ready. That's not easy for
all possible implementations. Drop it to simplify things.
Reviewed By: sfilipco
Differential Revision: D24399501
fbshipit-source-id: 4d85f77c99208cda30b2a543a0bb5b295f49a65c
Summary: There were 2 prepare_filesystem_sync. Unify them into one implementation.
Reviewed By: sfilipco
Differential Revision: D24399513
fbshipit-source-id: 80d009c33b7f23dc2c4225da6fd0fb09589ba061
Summary: More general purposed type for Syncable{IdDag,IdMap}.
Reviewed By: sfilipco
Differential Revision: D24399502
fbshipit-source-id: 0599db6dd07fe3d430458f86a33a9144d850fca1
Summary: This makes it more generic.
Reviewed By: sfilipco
Differential Revision: D24399493
fbshipit-source-id: 8a1d0a13dd29989b17fe3ef1497b10b6fa0629d6
Summary: Similar to IdDag change, move impls to separate files.
Reviewed By: sfilipco
Differential Revision: D24399508
fbshipit-source-id: 575b6e7194677b67b6755b0a30ae7d014d498b10
Summary:
The lock, reload, mutate, persist pattern is general. It can be used for IdMap
too.
Reviewed By: sfilipco
Differential Revision: D24399512
fbshipit-source-id: d25e51ba735061ca101101d75aff95deb88b1d36
Summary:
Now `build_segments_persistent` and `build_segments_volatile` are the same.
Just keep one of them.
Reviewed By: sfilipco
Differential Revision: D24399511
fbshipit-source-id: a9f1ac920cdf5b448bd99bf9b6d4ca4160ba0304
Summary:
Previously, we keep the last high level segment per level in memory, and
drop it on disk. When we cross the memory / disk boundary, we had to
maintain such properties carefully. That was needed because some DAG
algorithms rely on complete high level segments.
Now that no DAG algorithms depend on such properties, let's just drop
the logic adding the last segment back to simplify the code.
This removes the need of building segments after open() and sync().
Reviewed By: sfilipco
Differential Revision: D24399515
fbshipit-source-id: 4c640d9aa03c050fcd97f70ee386e32d3a8ee26d
Summary:
This makes the algorithm a bit more robust. Now none of the DAG algorithms
depend on high-level segments are complete and cover all low-level segments.
This also removes constraints. For example, SyncableIdDag can now just
deref() to the normal IdDag for queries without worrying about correctness.
Reviewed By: sfilipco
Differential Revision: D24399503
fbshipit-source-id: e6a91010cff82264cf423e2f24dee1d372822ef6
Summary:
They depend on high-level segments covering low-level segments, which
adds extra complexities. Remove them to simplify logic.
Reviewed By: sfilipco
Differential Revision: D24399509
fbshipit-source-id: 56a8e06c263107d1da4d6754b884ce51e18e30bf
Summary: The panics can happen when the input sets are out of range.
Reviewed By: kulshrax
Differential Revision: D24191789
fbshipit-source-id: efbcbd7f6f69bd262aa979afa4f44acf9681d11e
Summary:
Some sort of serialization for the Dag is useful for saving the IdDag produced
by offline jobs load that when a mononoke server starts.
Reviewed By: quark-zju
Differential Revision: D24096964
fbshipit-source-id: 5fac40f9c10a5815fbf5dc5e2d9855cd7ec88973
