Summary:
This was lost somehow. I probably incorrectly resolved some conflict when
rebasing a previous change.
Reviewed By: quark-zju
Differential Revision: D27146022
fbshipit-source-id: 13bb0bb3df565689532b2ab5299cd757f278f26e
Summary:
Finding a parent that was previously found signals that we want to assign
that changeset sooner if it was not already assigned.
Reviewed By: quark-zju
Differential Revision: D27092205
fbshipit-source-id: ed39a91460ff2f91a458236cdab8018341ec618b
Summary:
Seeding fbsource I found that loading the commits from sql took longer than I
was expecting, around 90 minutes where I was expecting around 10 miuntes.
I added more logging to validate that commits were actively loaded rather
than something being stuck.
Reviewed By: krallin
Differential Revision: D27084739
fbshipit-source-id: 07972707425ecccd4458eec849c63d6d9ccd923d
Summary:
Pretty big bug here with the "Overlay" when we are updating both stores. It
turns out that we don't really want a standard Overlay. We want the loaded
iddag to operate with the Ids in the shared IdMap and we want whatever is
updates to use the in process IdMap. The problem we have with the overlay is
that the shared IdMap may have more data than the in process IdMap. The shared
IdMap is always updated by the tailer, after all. This means that when we query
the overlay, we may get data from the shared store even if this is the first
time we are trying to update a changeset for the current process.
The solution here is to specify which vertexes are fetched from either store.
Reviewed By: quark-zju
Differential Revision: D27028367
fbshipit-source-id: e09f003d94100778eabd990724579c84b0f86541
Summary:
Using the generic load function from SegmentedChangelogManager. This is the
config SegmentedChangelog that is consistent with the specified configuration.
I wanted to have another look at ArcSwap to understand if
`Arc<ArcSwap<Arc<dyn SegmentedChangelog>>>` was the type that it was
recommending for our situation and indeed it is.
Reviewed By: quark-zju
Differential Revision: D27028369
fbshipit-source-id: 7c601d0c664f2be0eef782700ef4dcefa9b5822d
Summary:
Scuba stats provide a lot of context around the workings of the service.
The most interesting operation for segmented changelog is the update.
Reviewed By: krallin
Differential Revision: D26770846
fbshipit-source-id: a5250603f74930ef4f86b4167d43bdd1790b3fce
Summary:
STATS!!!
Count, success, failure, duration. Per instances, per repo.
I wavered on what to name the stats. I wondered whether it was worth being more
specific that "mononoke.segmented_changelog.update" with something like
"inprocess". In my view the in process stats are more important than the tailer
stats because the tailer is more simple and thus easier to understand. So I add
extra qualifications to the tailer stats and keep the name short for inprocess
stats.
Reviewed By: krallin
Differential Revision: D26770845
fbshipit-source-id: 8e02ec3e6b84621327e665c2099abd7a034e43a5
Summary: Currently unused. Will add stats the reference it.
Reviewed By: krallin
Differential Revision: D26770847
fbshipit-source-id: d5694cd221c90ba3adaf89345ffeb06fa46b9e7b
Summary:
I am not sure why the integration tests didn't fail for this one. I know that
a similar issue was caught last week. Probably one of those cases where not
all tests ran. Anyway. SegmentedChangelogManager requires bookmarks now.
It's not going to use them with the way to SegmentedChangelog is built. Using
the bookmarks needs another code change.
I noticed this because it was failing the Tailer. It will crash Mononoke too.
Long story on why the tailer uses this codepath. Needless to say, we don't want
Mononoke crashing so FIX :)
Reviewed By: quark-zju
Differential Revision: D26962608
fbshipit-source-id: 6efafc67f0816792b841af2cc456edc0cc579460
Summary:
Using a more specific name. Looking to differentiate between tailer update
and in process dag update.
Reviewed By: krallin
Differential Revision: D26770844
fbshipit-source-id: b35e6e705a0bfac6289c70a8e8e8cb9ba38a8d99
Summary:
Our production setup has an OnDemandUpdateSegmentedChangelog that gets updated
in various ways. With a setup where the dag is reloaded completely from saves,
we need a factory for the OnDemandUpdateSegmentedChangelog.
SegmentedChangelogManager takes the role of being the factory for our
production Dags.
At some point we will remove the SegmentedChangelog implementation for Manager.
Reviewed By: krallin
Differential Revision: D26708173
fbshipit-source-id: b3d8ea612b317af374f2c0ce6d7c512e3b09b2d2
Summary:
The manager was added as a high level abstraction for storing and loading a
SegmentedChangelog. It worked well when we had one configuration for
SegmentedChangelog. The problem now is that SegmentedChangelog has various
configurations. Storing and loading is an asymetric operation.
In contexts where we do storing we want to have used a specific configuration,
one that operates on an owned dag and has an IdMap that writes to the database.
Then, when running on the server we never store, our writes to the idmap are
in process only and the iddag is wrapped in layers that keep it up to date.
The manager would have to be too complicated to handle all these scenarios.
The solution here is to simplify the manager to cater to the server use case
and inline the logic for the saves where it is used (seeder and tailer).
Reviewed By: krallin
Differential Revision: D26921451
fbshipit-source-id: aedf4acf4bc8371a5d0b249f8bccd9447e85ae0a
Summary:
At the same time remove SqlIdMapFactory. Consolidate the details surrounding
building the IdMap in this factory by moving the logic for caching and in
memory construction from the Manager to the factory.
Reviewed By: krallin
Differential Revision: D26708177
fbshipit-source-id: a6a7f6270c2508adf85f529eef2c75653d002cd0
Summary:
Consolidating on the SegmentedChangelog suffix for the structures in the
`segmented_changelog` crate.
Reviewed By: quark-zju
Differential Revision: D26891996
fbshipit-source-id: 75192bed9cc073adfe7b82ac2b60516ac6629b76
Summary:
Consolidating on the SegmentedChangelog suffic for the structures in the
`segmented_changelog` crate.
Reviewed By: quark-zju
Differential Revision: D26892000
fbshipit-source-id: 47c6ece8aa7ef13e3ea51bbe558655e3f61fdedf
Summary:
Consolidating on the SegmentedChangelog suffix for the structures in the
`segmented_changelog` crate.
Reviewed By: quark-zju
Differential Revision: D26892003
fbshipit-source-id: ad1ccb8c359e7cd5b58d053aa13ed908252988b0
Summary:
Consolidating on the SegmentedChangelog suffix for the structures in the
`segmented_changelog` crate.
Reviewed By: quark-zju
Differential Revision: D26891998
fbshipit-source-id: 86576a029f851e0ac4a6d6600a8839289c9f1f93
Summary:
Consolidating on the SegmentedChangelog suffix for the structures in the
`segmented_changelog` crate.
Reviewed By: quark-zju
Differential Revision: D26892002
fbshipit-source-id: df52027a7c20684c0d46b7adc80692d262b669d4
Summary:
The macro helps with implementing SegmentedChangelog interface for the
structures that rely on another SegmentedChangelog.
Reviewed By: quark-zju
Differential Revision: D26892001
fbshipit-source-id: 6e5f1f04b47f814cf7ed6fd67f4797c5270ba701
Summary:
Consolidating on SegmentedChangelog for the structures in the
`segmented_changelog` crate. We treat these structures as a specific kind of
dag and we name them specifically.
The `dag` crate can have the Dag structures. The `dag` crate generalizes the
graph concept. Dag for generalization, SegmentedChangelog for specific use.
The migration on the DB is simple. We will stop the tailer processes and copy
the data from `segmented_changelog_bundle` to `segmented_changelog_version`.
We will then update the jobs to an ephemeral package that uses
`segmented_changelog_version`. We will remove the old table a week later.
Reviewed By: quark-zju
Differential Revision: D26891997
fbshipit-source-id: e0061973942defa09493b4d23c89d2aaed40825a
Summary:
AsyncVfs provides async vfs interface.
It will be used in the native checkout instead of current use case that spawns blocking tokio tasks for VFS action
Reviewed By: quark-zju
Differential Revision: D26801250
fbshipit-source-id: bb26c4fc8acac82f4b55bb3f2f3964a6d0b64014
Summary: Now the queries macros are asynced, lets do the same with the Transaction api exposed from them.
Reviewed By: krallin
Differential Revision: D26730195
fbshipit-source-id: 278753a5d0401f602ce50519138164bb5e49d550
Summary: Migrate to the std futures version of sql::queries!
Reviewed By: krallin
Differential Revision: D26700360
fbshipit-source-id: 9ed2664d522bde8d0e923142357ca876a7de2613
Summary:
Async the query macros. This change also migrates most callsites, with a few more complicated ones handle as separate diffs, which temporarily use sql01::queries in this diff.
With this change the query string is computed lazily (async fn/blocks being lazy) so we're not holding the extra memory of query string as well as query params for quite as long. This is of most interest for queries doing writes where the query string can be large when large values passed (e.g. Mononoke sqlblob blobstore )
Reviewed By: krallin
Differential Revision: D26586715
fbshipit-source-id: e299932457682b0678734f44bb4bfb0b966edeec
Summary:
This diffs add a layer of indirection between fbinit and tokio, thus allowing
us to use fbinit with tokio 0.2 or tokio 1.x.
The way this works is that you specify the Tokio you want by adding it as an
extra dependency alongside `fbinit` in your `TARGETS` (before this, you had to
always include `tokio-02`).
If you use `fbinit-tokio`, then `#[fbinit::main]` and `#[fbinit::test]` get you
a Tokio 1.x runtime, whereas if you use `fbinit-tokio-02`, you get a Tokio 0.2
runtime.
This diff is big, because it needs to change all the TARGETS that reference
this in the same diff that introduces the mechanism. I also didn't produce it
by hand.
Instead, I scripted the transformation using this script: P242773846
I then ran it using:
```
{ hg grep -l "fbinit::test"; hg grep -l "fbinit::main" } | \
sort | \
uniq | \
xargs ~/codemod/codemod.py \
&& yes | arc lint \
&& common/rust/cargo_from_buck/bin/autocargo
```
Finally, I grabbed the files returned by `hg grep`, then fed them to:
```
arc lint-rust --paths-from ~/files2 --apply-patches --take RUSTFIXDEPS
```
(I had to modify the file list a bit: notably I removed stuff from scripts/ because
some of that causes Buck to crash when running lint-rust, and I also had to add
fbcode/ as a prefix everywhere).
Reviewed By: mitrandir77
Differential Revision: D26754757
fbshipit-source-id: 326b1c4efc9a57ea89db9b1d390677bcd2ab985e
Summary:
This diff rollouts V2 of autocargo in an atomic way so there are quite a few things done here.
Arc lint support:
V1 used to be part of the default fbsource `arc lint` engine, but since V2 calls buck it must live in a separate lint engine. So this diff:
- Adds running `autocargo` as part of `arc lint-rust`
Mergedriver update:
- Mergedriver used in resolving conflicts on commits is now pointing to V2
- It handles files in `public_autocargo/` directories in addition to the ones containig generation preamble
Including regeneration results of running `common/rust/cargo_from_buck/bin/autocargo`. All the differences are accounted for:
- Some sections and attributes are removed as they can be autodiscovered by Cargo (like `lib.path = "src/lib.rs"` or empty [lib] section)
- "readme" attribute is properly defined as relative to Cargo.toml location rather than as hardcoded string
- "unittest = false" on a Buck rule propagates as "test = false; doctest = false" to Cargo
- "rusqlite" is not special-cased anymore, so the "budled" feature will have to be enabled using custom configuration if required by the project (for rust-shed in order to not break windows builds a default feature section was added)
- Files generated from thrift_library rules that do not support "rust" language are removed
- Custom .bzl rules that create rust artifacts (like `rust_python_extension`) are no longer ignored
Others:
- Changed `bin/cargo-autocargo` to be a wrapper for calling V2 via `cargo autocargo`
- Updated following files to use V2:
- `common/rust/tools/reindeer/version-bump`
- `remote_execution/rust/setup.sh`
- Removed few files from V1 that would otherwise interfere with V2 automatic regeneration/linting/testing
Reviewed By: zertosh
Differential Revision: D26728789
fbshipit-source-id: d1454e7ce658a2d3194704f8d77b12d688ec3e64
Summary:
This dag periodically reloads the dag from storage.
It currently loads a simple dag that has no update logic because that is what
the manager returs. It's not relevant for this code.
This is probably the last piece before we refactor construction to take a
SegmentedChangelogConfig. To be seen how much will be strict types and how much
will be Arc<dyn SegmentedChangelog>.
Reviewed By: krallin
Differential Revision: D26681458
fbshipit-source-id: 6056d00db6f25616e8158278702f9f4120b92121
Summary: There were no unit test for SegmentedChangelogManager so I added one.
Reviewed By: krallin
Differential Revision: D26681459
fbshipit-source-id: 40ceefe7b89043ae6d2c4d31a2adf504245161fb
Summary:
A placeholder for convenience functions.
Right not it has a proxy for the head of the dag.
Reviewed By: krallin
Differential Revision: D26681457
fbshipit-source-id: 6856abbf2685407f96701ea5a508342373503360
Summary:
An OnDemandUpdateDag can now track a bookmark. Every given period it will
query the changeset of the bookmark and incrementally build the dag.
Reviewed By: krallin
Differential Revision: D26656765
fbshipit-source-id: 95057863b5201f9632c654be5544922c7538f974
Summary:
For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present.
Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is.
The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed.
Reviewed By: ikostia
Differential Revision: D26618363
fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
Summary:
Simple test that can give us an intuition for how the ConcurrentMemIdMap should
perform.
Reviewed By: krallin
Differential Revision: D26601378
fbshipit-source-id: ae8f2ada6fc08eef806f3ece72a6c1c2f011ac32
Summary: I removed tokio-compat yesterday but this landed at the same time and uses it.
Reviewed By: mitrandir77, StanislavGlebik
Differential Revision: D26605246
fbshipit-source-id: 189f485bc8bc3018abb3e9290953eba14bd178de
Summary:
Adding a new configuration that instantiates SegmentedChangelog by downloading
a dag from a prebuilt blob. It then updates in process.
Reviewed By: krallin
Differential Revision: D26508428
fbshipit-source-id: 09166a3c6de499d8813a29afafd4dfe19a19a2a5
Summary:
I am not sure if this too abstract. It might be. This however has separation of
concerns :)
The goal here is to end up with an in memory IdMap that we write to and read
from first. For things that are not found in the in memory IdMap we fall back
to the SqlIdMap. We'll end up with something like:
`OverlayIdMap(ConcurrentMemIdMap, SqlIdMap)`
Reviewed By: quark-zju
Differential Revision: D26417642
fbshipit-source-id: b2b310306db4dc9fc3427bbf50b19366160882a9
Summary:
The `MemIdMap` is not a valid `IdMap` implementation because it takes `&mut
self` when doing inserts. Wrapping the IdMap in a `RwLock` allows us to
implement the `IdMap` trait.
Reviewed By: krallin
Differential Revision: D26417643
fbshipit-source-id: cb5e3513841fa1dd7c8b8004ce7b2fe1467983d7
Summary:
The on demand update code we have is the most basic logic that we could have.
The main problem is that it has long and redundant write locks. This change
reduces the write lock strictly to the section that has to update the in memory
IdDag.
Updating the Dag has 3 phases:
* loading the data that is required for the update;
* updating the IdMap;
* updating the IdDag;
The Dag can function well for serving requests as long as the commits involved
have been built so we want to have easy read access to both the IdMap and the
IdDag. The IdMap is a very simple structure and because it's described as an
Arc<dyn IdMap> we push the update locking logic to the storage. The IdDag is a
complicated structure that we ask to update itself. Those functions take
mutable references. Updating the storage of the iddag to hide the complexities
of locking is more difficult. We deal with the IdDag directly by wrapping it in
a RwLock. The RwLock allows for easy read access which we expect to be the
predominant access pattern.
Updates to the dag are not completely stable so racing updates can have
conflicting results. In case of conflics one of the update processes would have
to restart. It's easier to reason about the process if we just allow one
"thread" to start an update process. The update process is locked by a sync
mutex. The "threads" that fail the race to update are asked to wait until the
ongoing update is complete. The waiters will poll on a shared future that
tracks the ongoing dag update. After the update is complete the waiters will go
back to checking if the data they have is available in the dag. It is possible
that the dag is updated in between determining that the an update is needed and
acquiring the ongoing_update lock. This is fine because the update building
process checks the state of dag before the dag and updates only what is
necessary if necessary.
Reviewed By: krallin
Differential Revision: D26508430
fbshipit-source-id: cd3bceed7e0ffb00aee64433816b5a23c0508d3c
Summary:
This structure is going to be useful to implement the SegmentedChangelog
functionlity for the OnDemandDag as we move forward with separate objects for
the iddag and the idmap rather than a direct dependency on a Dag object.
Reviewed By: quark-zju
Differential Revision: D26508429
fbshipit-source-id: 9116f1c82d301e8e5b726966abd2add2e32765d6
Summary:
Moving all update logic to `dag::update`.
Additional minor changes: removing Dag::build and spliting build_incremental
around the mutable update section of iddag.
Reviewed By: krallin
Differential Revision: D26508427
fbshipit-source-id: 984259d2f199792fcf0635dd3100ec39260fd3ed
Summary:
This is going to be useful when we update the InProcessDag on the server. As
opposed to taking a lock for everything we will be able to take a short lock
to update the IdMap and then a short lock to update the IdDag.
Reviewed By: krallin
Differential Revision: D26417621
fbshipit-source-id: 43f342d384f1be80dcfe721de659ac3ce9dd0e7b
Summary:
The earlier diffs in this stack have removed all our dependencies on the Tokio
0.1 runtime environment (so, basically, `tokio-executor` and `tokio-timer`), so
we don't need this anymore.
We do still have some deps on `tokio-io`, but this is just traits + helpers,
so this doesn't actually prevent us from removing the 0.1 runtime!
Note that we still have a few transitive dependencies on Tokio 0.1:
- async-unit uses tokio-compat
- hg depends on tokio-compat too, and we depend on it in tests
This isn't the end of the world though, we can live with that :)
Reviewed By: ahornby
Differential Revision: D26544410
fbshipit-source-id: 24789be2402c3f48220dcaad110e8246ef02ecd8
Summary:
The changes (and fixes) needed were:
- Ignore rules that are not rust_library or thrift_library (previously only ignore rust_bindgen_library, so that binary and test dependencies were incorrectly added to Cargo.toml)
- Thrift package name to match escaping logic of `tools/build_defs/fbcode_macros/build_defs/lib/thrift/rust.bzl`
- Rearrange some attributes, like features, authors, edition etc.
- Authors to use " instead of '
- Features to be sorted
- Sort all dependencies as one instead of grouping third party and fbcode dependencies together
- Manually format certain entries from third-party/rust/Cargo.toml, since V2 formats third party dependency entries and V1 just takes them as is.
Reviewed By: zertosh
Differential Revision: D26544150
fbshipit-source-id: 19d98985bd6c3ac901ad40cff38ee1ced547e8eb
Summary:
Autocargo V2 will use a more structured format for autocargo field
with the help of `cargo_toml` crate it will be easy to deserialize and handle
it.
Also the "include" field is apparently obsolete as it is used for cargo-publish (see https://doc.rust-lang.org/cargo/reference/manifest.html#the-exclude-and-include-fields). From what I know this might be often wrong, especially if someone tries to publish a package from fbcode, then the private facebook folders might be shipped. Lets just not set it and in the new system one will be able to set it explicitly via autocargo parameter on a rule.
Reviewed By: ahornby
Differential Revision: D26339606
fbshipit-source-id: 510a01a4dd80b3efe58a14553b752009d516d651
Summary:
One of the primary use cases for hash_to_location is translating user provided
hashes. It is then perfectly valid for the hashes that are provided to not
exist. Where we would previously return an error for the full request if a
hash was invalid, we now omit the hash from the response.
Reviewed By: quark-zju
Differential Revision: D26389472
fbshipit-source-id: c59529d43f44bed7cdb2af0e9babc96160e0c4a7
Summary:
Get the graph location of a given commit identifier.
The client using segmented changelog will have only a set of identifiers for
the commits in the graph. The client needs a way to translate user input to
data that it has locally. For example, when checking out an older commit by
hash the client will have to retrieve a location to understand the place in the
graph of the commit.
Reviewed By: quark-zju
Differential Revision: D26289623
fbshipit-source-id: 4192d91a4cce707419fb52168c5fdff53ac3a9d0
Summary:
If `location_to_changeset_id` somehow does not get a vector with one element
it will panic and our server will stop immediately. Not great for a server.
Changes are low but we already return `anyhow::Result` so no big pain here.
Reviewed By: quark-zju
Differential Revision: D26180417
fbshipit-source-id: 6986f3fdd0b34f7c2606162bc35aacb9857ea04c
Summary:
We had individual fields for location components. This change will make it
easier for people to read through the code. This completes the integration
of the Location struct.
Reviewed By: quark-zju
Differential Revision: D26162272
fbshipit-source-id: 76259578430bac88317afb1935f63e06b6e8284e
Summary:
This is removing `edenapi::CommitLocation` in order to use
`dag_types::Location`.
First, `edenapi::CommitLocation` has a `count` field and `dag_types::Location`
does not. I find `count` to be difficult to attach to a more general structure.
In practice `edenapi::CommitLocation` is replaced by `CommitLocationToHashRequest`.
On top of the request we have the batch object: `CommitLocationToHashRequestBatch`.
Second, `edenapi::CommitLocation` did not have Wire types, the new structures do.
Reviewed By: quark-zju
Differential Revision: D26159865
fbshipit-source-id: f2508e123e11988726868c7f439a2ed186afce5c
Summary:
In the public API at least. A public method will consume the builder. If some
code wants to call multiple methods using the same configuration, they can
safely clone the builder to get a second instance.
SegmentedChangelogBuilder needs to pass references internally to build
individual components otherwise it would have to clone itself excessively.
This pattern leaked towards public methods too. Some tests use this builder too
and they use some crate public methods that need to be defined using
references. I don't know if we should remove that dependency. Anyway, the
Builder is hopefully easier to use now.
Reviewed By: quark-zju
Differential Revision: D26152066
fbshipit-source-id: 63285e200d8e9fde06fede03773b7d4c02e9cea7
Summary:
Caching for the IdMap to speed things up.
Values for a key do not change. The IdMap is versioned for a given Repository.
We use the version of the IdMap in the generation of the cache keys. We set the
"site version" to be the IdMap version.
Reviewed By: krallin
Differential Revision: D26121498
fbshipit-source-id: 7e82e40b818d1132a7e86f4cd7365dd38056348e
Summary:
We have fallback logic to go to the leader if the data we want is missing in
the replica, but right now it's backwards so we go to the leader to find data
we actually *did* find in the replica (and we don't go to the leader for
missing data).
Reviewed By: sfilipco
Differential Revision: D26103898
fbshipit-source-id: 535abab2a3093165f1d55359d102a7a7cb542a9c
Summary:
Lots of generated code in this diff. Only code change was in
`common/rust/cargo_from_buck/lib/cargo_generator.py`.
Path/git-only dependencies (ie `mydep = { path = "../foo/bar" }`) are not
publishable to crates.io. However, we are allowed to specify both a path/git
_and_ a version. When building locally, the path/git is chosen. When publishing,
the version on crates.io is chosen.
See https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#multiple-locations .
Note that I understand that not all autocargo projects are published on crates.io (yet).
The point of this diff is to allow projects to slowly start getting uploaded.
The end goal is autocargo generated `Cargo.toml`s that can be `cargo publish`ed
without further modification.
Reviewed By: lukaspiatkowski
Differential Revision: D26028982
fbshipit-source-id: f7b4c9d4f4dd004727202bd98ab10e201a21e88c
Summary:
When we tried to update to Tokio 0.2.14, we hit lots of hangs. Those were due
to incompatibilities between Tokio 0.2.14 and Futures 1.29. We fixed some of
the bugs (and others had been fixed and were pending a release), and Futures
1.30 have now been released, which unblocks our update.
This diff updates Tokio accordingly (the previous diff in the stack fixes an
incompatibility).
The underlying motivation here is to ease the transition to Tokio 1.0.
Ultimately we'll be pulling in those changes one or way or another, so let's
get started on this incremental first step.
Reviewed By: farnz
Differential Revision: D25952428
fbshipit-source-id: b753195a1ffb404e0b0975eb7002d6d67ba100c2
Summary:
This feature is useful for testing time-dependent stuff (e.g. it
allows you to stop/forward time). It's already included in the buck build.
Reviewed By: SkyterX
Differential Revision: D25946732
fbshipit-source-id: 5e7b69967a45e6deaddaac34ba78b42d2f2ad90e
Summary: When scrubbing repos it is preferable to scrub newest data first. This diff adds Direction::NewestFirst to bulkops for use in scrubbing and updates existing call sites to Direction::OldestFirst so as not to change behaviour
Reviewed By: StanislavGlebik
Differential Revision: D25742279
fbshipit-source-id: 363a4854b14e9aa970b2c1ec491dcaccac7a6ec9
Summary: There is only one implementation of the trait so remove it and use that impl directly. Removing the trait makes it simpler to work on bulkops in the rest of this stack.
Reviewed By: farnz
Differential Revision: D25804021
fbshipit-source-id: 22fe797cf87656932d383ae236f2f867e788a832
Summary:
Comments for why we don't need a lock when updating the SqlIdMap with multiple
writers. Structure can definitely be improved but I'll live with this for a
short time.
No fundamental change in logic. I added extra checks to the insert function and
changed from an optimistic insert race logic to a pessimistic version. I
explain in the comments that it's to have an easier time reasoning about what
happens and that theoretically doesn't matter.
Reviewed By: quark-zju
Differential Revision: D25606290
fbshipit-source-id: ea21915fc797fe759b3fe481e8ad9e8cb594fb6a
Summary:
The end goal is to have clients using a sparse IdMap. There is still some work
to get there though. In the mean time we can test repositories that don't use
any revlogs. The current expections for those repositories are that they have
a full idmap locally.
Reviewed By: quark-zju
Differential Revision: D25075341
fbshipit-source-id: 52ab881fc9c64d0d13944e9619c087e0d4fb547c
Summary:
The client dag cannot currently be instantiated with a sparse idmap (aka
universal commit idmap). Is should be usable with a full idmap. To test
repositories that use segmented changelog exclusively we add the capability of
cloning the full idmap.
I currently see StreamCloneData as an experiment. I am open to suggestions
around what structure we should have for the regular long term clone endpoint.
That said, I am leaning towards converting clone_data to return
StreamCloneData. Overall, Segmented Changelog has a few knobs that influence
how big the IdMap ends up being so the code that is more flexible will be more
useful long term. To add to that, we transform data higher in the stack using
streaming and this data does similar fetching, it seems that we should have a
stream idmap exposed by clone_data.
Reviewed By: quark-zju
Differential Revision: D24966338
fbshipit-source-id: 019b363568e3191280bd5ac09fc15062711e5523
Summary: Convert `ChangesetFetcher` to new type futures
Reviewed By: StanislavGlebik
Differential Revision: D25244213
fbshipit-source-id: 4207386d81397a930a566db008019bb8f31bf602
Summary: Basic observability for how the segmeted changelog update process is performing.
Reviewed By: krallin
Differential Revision: D25108739
fbshipit-source-id: b1f406eb0c862464b186f933d126e0f3a08144e4
Summary:
The update of the segmented changelog is light weight enough that we can
consider all repositories sharing a common tailer process. With all
repositories sharing a single tailer the the maintenance burden will be lower.
Things that I am particularly unsure about are: tailer configuration setup and
tailer structure. With regards to setup, I am not sure if this is more or less
than what production servers do to instantiate. With regards to structure, I
think that it makes a lot of sense to have a function that takes a single repo
name as parameter but the configuration setup has an influence on the details.
I am also unsure how important it is to paralelize the instantiation of the
blobrepos.
Finally, it is worth mentioning that the SegmentedChangelogTailer waits for
`delay` after an update finishes rather than on a period. The benefit is that
we don't have large updates taking down a process because we schedule the same
large repo update too many timer. The drawback is that scheduling gets messed
up over time and multiple repo updates can end up starting at the same time.
Reviewed By: farnz
Differential Revision: D25100839
fbshipit-source-id: 5fff9f87ba4dc44a17c4a7aaa715d0698b04f5c3
Summary:
This diff prepares the Mononoke codebase for composition-based extendability of
`ScubaSampleBuilder`. Specifically, in the near future I will add:
- new methods for verbose scuba logging
- new data field (`ObservabilityContext`) to check if verbose logging should
be enabled or disabled
The higher-level goal here is to be able to enable/disable verbose Scuba
logging (either overall or for certain slices of logs, like for a certain
session id) in real time, without restarting Mononoke. To do so, I plan to
expose the aforementioned verbose logging methods, which will run a check
against the stored `ObservabilityContext` and make a decision of whether the
logging is enabled or not. `ObservabilityContext` will of course hide
implementation details from the renamed `ScubaSampleBuilderExt`, and just provide a yes/no
answer based on the current config and sample fields.
At the moment this should be a completely harmless change.
Reviewed By: krallin
Differential Revision: D25211089
fbshipit-source-id: ea03dda82fadb7fc91a2433e12e220582ede5fb8
Summary:
Under this configuration SegmentedChangelog Dags (IdDag + IdMap) are always
downloaded from saves. There is no real state kept in memory.
It's a simple configuration and somewhat flexible with treaks to blobstore
caching.
Reviewed By: krallin
Differential Revision: D24808330
fbshipit-source-id: 450011657c4d384b5b42e881af8a1bd008d2e005
Summary:
Constructs and returns `CloneData<ChangesetId>`. This object can then be used
to bootstrap a client dag that speaks bonsai commits.
Short term we are going to be using this data in the Mercurial client which
doesn't use bonsai. Hg MononokeRepo will convert it.
Long term we may decide that we want to download cached artifacts for
CloneData. I don't see an issue getting there, I see this as a valid path
forward that cuts down on the configuration required to get to the cached
artifacts. All that said, I think that using whatever dag is available in
memory would be a viable production option.
Reviewed By: krallin
Differential Revision: D24717915
fbshipit-source-id: 656924abb4bbfa1a11431000b6ca6ed2491cdc74
Summary: The SegmentedChangelogManager abstracts saving and loading Dags. This is currently used in the tailer and seeder processes. It will also be used to load dags while the server is running.
Reviewed By: krallin
Differential Revision: D24717925
fbshipit-source-id: 30dff7dfc957f455be6cf733b20449c804511b43
Summary:
MySQL doesn't like that the idmap table is renamed to `inner`. For good reason,
inner is a keyword, best to rename it.
Reviewed By: ahornby
Differential Revision: D24568914
fbshipit-source-id: 7a3790e835931b29658c7652cc89069c6b9b5bab
Summary:
I avoided this function because it interacts in a weird ways with dependencies.
At this point I am no longer concerned about that and it can help us simplify
some code.
Looking ahead I think that we will refactor things into having fewer
dependencies.
Reviewed By: krallin
Differential Revision: D24555935
fbshipit-source-id: 994b25d90da491bb5cc593b6c33085790c4fb322
Summary:
The command reads the last SegmentedChangelog that was saved for a repository
and updates head to match a given bookmark (master).
Right now this is just a command that works on one repository. Follow up
changes will look at deployment options and handling multiple repositories.
Reviewed By: krallin
Differential Revision: D24516438
fbshipit-source-id: 8f04f9426c2f2d7748c5363d2dbdf9f3acb79ddd
Summary:
I initially saw the incremental build as something that would be run in places
that had IdMap and IdDag stored side by side in process. I am reconsidering
to use incremental build in the tailing process to keeps Segmented Changelog
artifacts up to date.
Since we update the IdMap before we update the IdDag, it is likely that we
will have runs that only update the IdMap and fail to update IdDags. This diff
adds a mechanism for the IdDag to catch up.
Reviewed By: krallin
Differential Revision: D24516440
fbshipit-source-id: 3a99248451d806ae20a0ba96199a34a8a35edaa4
Summary:
Nice to have things for debugging.
This isn't an exhaustive list of places that we could add context too. I'll
probably look to complete the list after the current changes are done.
Reviewed By: krallin
Differential Revision: D24516437
fbshipit-source-id: 7f29e7afde5a5918aea419181d786f48da9b8b14
Summary:
The general goal is to allign segmented changelog blobstore usage with the
general pattern in Mononoke.
Reviewed By: quark-zju
Differential Revision: D24605796
fbshipit-source-id: 808985609f74ebc45f3fcc57583e55f3af9bce1d
Summary:
From an OSS perspective, I think that the log tables have a place. However for
daily use perspective, next to scuba they don't add much except retention and
instead feel more heavy weight to manage. This change probably simplifies
things and makes the Segmented Changelog component easier to maintain.
Reviewed By: krallin
Differential Revision: D24213548
fbshipit-source-id: 48a4ea57e3f3911c3bf82b0cc51f118d72119e19
Summary: looks like we got land time conflict
Reviewed By: krallin
Differential Revision: D24196362
fbshipit-source-id: 27da83a2f86cc7fe5f59fe583d4b719f69df0248
Summary:
Mononoke command for running the SegmentedChangelogSeeder for an existing
repository. The result is going to be a new IdMap version in the metadata
store and a new IdDag stored in the the blobstore resulting in a brand new
SegmentedChangelog bundle.
Reviewed By: krallin
Differential Revision: D24096963
fbshipit-source-id: 1eaf78392d66542d9674a99ad0a741f24bc2cb1b
Summary:
The SegmentedChangelogSeeder has the role of constructing a new IdMap for a
given repository. That would happen when a repository is onboarded or when
algorithm improvements are made.
This change comes with small refactoring. We had the Dag which did a bit of
everything. Now the on_demand_update and the seeder functionalities are in
their separate files. The tests from `dag.rs` were moved to the `tests.rs` and
updated to use the seeder and on_demand_update structures.
`SegmentedChangelogSeeder::run` is the main logic added in this diff.
Reviewed By: quark-zju
Differential Revision: D24096965
fbshipit-source-id: 0f655e8c226ca0051f3e925342e92b1e7979aab2
Summary:
The IdDagStore provides the ability to save and later load prebuilt instances
of the IdDag.
This is going to be used in the clone API where we send one of these blobs to
the client. It is also going to be used by servers starting up.
Right now the serialization is naive, relying on serde::Serialize. The key
schema would provide the means for evolving the serialization format in cases
where we would require breaking changes.
Reviewed By: quark-zju
Differential Revision: D24096967
fbshipit-source-id: 2c883e5e82c05bec03c429c3c2a2d545170a8c05
Summary:
This IdMapVersionStore determines which is the latest IdMapVersion that commit
"tailing" processes should use when building new Dag bundles. The "seed"
process will update the versions of the IdMap. The plan for the "seed" process
is to write a new IdMap version to Sql then update the store with a new entry.
New "tailer" processes will then start to use the newly built IdMapVersion.
The tailing processes that will build fresh IdDags for general consumption.
These IdDags will be used by the clone operation. These dags will also be used
by servers instances spinning up.
DagBundles specify (id dag version, id map vession). This pair specified a
prebuilt Segmented Changelog that is ready to be loaded.
Reviewed By: quark-zju
Differential Revision: D24096968
fbshipit-source-id: 413f49ed185a770a73afd17dfbc952901ab53b42
Summary:
The version is going to be used to seamlessly upgrade the IdMap. We can
generate the IdMap in a variety of ways. Naturally, algorithms for generating
the IdMap may change, so we want a mechanism for updating the shared IdMap.
A generated IdDag is going to require a specific IdMap version. To be more
precise, the IdDag is going to specify which version of IdMap it has to be
interpreted with.
Reviewed By: quark-zju
Differential Revision: D23501158
fbshipit-source-id: 370e6d9f87c433645d2a6b3336b139bea456c1a0
Summary:
Separate the operational bits of the IdMap from the core SegmentedChangelog
requirements.
I debaded whether it make sense to add repo_id to SqlIdMap. Given the current
architecture I don't see a reason not to do it. On the contrary separating
two objects felt convoluted.
Reviewed By: quark-zju
Differential Revision: D23501160
fbshipit-source-id: dab076ab65286d625d2b33476569da99c7b733d9
Summary:
Planning to add a trait for core idmap functionality (that's just translating
cs_id to vertex and back). The current IdMap will then be an implementation of
that trait.
Reviewed By: quark-zju
Differential Revision: D23501159
fbshipit-source-id: 34e3b26744e4b5465cd108cca362c38070317920
Summary:
The Segmented Changelog must be built somewhere. One of the simplest deployments
of involves the on-demand update of the graph. When a commit that wasn't yet
processed is encountered, we sent it to processing along with all of it's
ancestors.
At this time not much attention was paid to the distinction of master commit
versus non-master commit. For now the expectation is that only commits from
master will exercise this code path. The current expectation is that clients
will only call location-to-hash using commits from master.
Let me know if there is an easy way to check if a commit is part of master.
Later changes will invest more in handling non-master commits.
Reviewed By: aslpavel
Differential Revision: D23456218
fbshipit-source-id: 28c70f589cdd13d08b83928c1968372b758c81ad
Summary:
This builders implements SqlConstruct and SqlConstuctFromMetadataDatabaseConfig
to make handling the Sql connection for IdMap consistent with what happens in
Mononoke in general.
Reviewed By: aslpavel
Differential Revision: D23456219
fbshipit-source-id: 6998afbbfaf1e0690a40be6e706aca1a3b47829f
