Summary: Need to add new quickcheck tests, verify that remove `Serialize` from `TreeEntry` is okay.
Reviewed By: kulshrax
Differential Revision: D23457777
fbshipit-source-id: aa94ed7aa81b41924eba4a8bd1bdc2c737365b77
Summary: Added sqlblob to the common blobstore tests to make sure it doesn't diverge from expectations for other stores.
Reviewed By: StanislavGlebik
Differential Revision: D24079254
fbshipit-source-id: 49ef1c372171a1594ba298c66d3473ef682d24cc
Summary:
Add CountedBlobstoreOps so that blobstore layers that need BlobstoreOps can still use counting
This unblocks adding sqlblob to blobstore-test in next diff in stack
Reviewed By: farnz
Differential Revision: D24079256
fbshipit-source-id: 6a6505aff8c8405353a1f10d79f6e6e08911228a
Summary: Add BlobstorePutOps so that blobstore layers that need BlobstorePutOps can still use PrefixBlob as a wrapper.
Reviewed By: farnz
Differential Revision: D24109298
fbshipit-source-id: 710571e6c30fa8a432d463eedfab5fcc0389baa3
Summary:
Add predicate based PutBehaviour logic to manifoldblob.
This will prevent overwrites of keys when in IfAbsent mode, and will generate useful logging in OverwriteAndLog and IsAbsent mode.
This change factors our part of the put logic to put_check_conflict, so that it can use re-used from each of the PutBehaviour cases.
Reviewed By: StanislavGlebik
Differential Revision: D24021170
fbshipit-source-id: d2e71afadada3d5e661634449108e6c9f8dc5907
Summary:
We don't have any Preserved entries anymore - now all preserved entries will be
rewritten with "noop" mapping.
This diff removes it completely
Reviewed By: mitrandir77, ikostia
Differential Revision: D24173538
fbshipit-source-id: f2d6238633cea8dc3c06f2e607b2abd76edfca6b
Summary: This state is going to be removed soon, so no need for tests anymore
Reviewed By: mitrandir77
Differential Revision: D24221363
fbshipit-source-id: 19dce04549ccbfe59255463a73e56c70f1c8bc4d
Summary: Just as with the previous diff, let's remove it from cross_repo_sync_test_utisl
Reviewed By: mitrandir77
Differential Revision: D24220618
fbshipit-source-id: 95c5ddc955f720101f6576b34e9787435b6deb4c
Summary:
From an OSS perspective, I think that the log tables have a place. However for
daily use perspective, next to scuba they don't add much except retention and
instead feel more heavy weight to manage. This change probably simplifies
things and makes the Segmented Changelog component easier to maintain.
Reviewed By: krallin
Differential Revision: D24213548
fbshipit-source-id: 48a4ea57e3f3911c3bf82b0cc51f118d72119e19
Summary:
This diff is a first step in a preparation for removing Preserved state from
CommitSyncOutcome. It does two things:
1) Remove unsafe_preserve_commit method and instead rewrite commit with noop
config version - this is exactly how we are going to do it in production
2) While there also select correct mover for validation - previously we were
always picking the current mover
I've also added a bunch of print statements, since it was tricky to debug what
was going on in the test.
Reviewed By: ikostia
Differential Revision: D24216168
fbshipit-source-id: 86e81aea61e638d93bdb33e7c9fd713f7b5e6c3b
Summary:
All the new entries in our mapping tables should have their version set. Let's
enforce it in the code
Reviewed By: ikostia
Differential Revision: D24217688
fbshipit-source-id: 95f01d8929a9c3a19b84434c91db6d08a6e5f863
Summary:
To give a bit of context - we are getting rid of Preserved state in
CommitSyncOutcome completely since it can be fully replaced with with
EquivalentWorkingCopyAncestor/RewrittenAs states.
ikostia@ did a similar change in D24142837 (2035a34a0e) to consider Preserved only if
mapping is None. However this diff diff it only for mapping.get() method. Let's do
the same for mapping.get_equivalent_working_copy().
Reviewed By: ikostia
Differential Revision: D24216455
fbshipit-source-id: f1f8d46263de54cb2e11d33b6c17f371b79e80f9
Summary: This was probably copied from backsyncer some time ago.
Reviewed By: StanislavGlebik
Differential Revision: D24198742
fbshipit-source-id: 3d8fad0ddc94185acd28ede7163b43424935830d
Summary:
This diff adds tests for `sync_merge` version-determination logic:
- when both parents were rewritten with the same version and its identical to the current one
- when both parents were rewritten with the same version and its different to the current one
- when both parents are Preserved
- when one parent is Preserved
Reviewed By: StanislavGlebik
Differential Revision: D24104680
fbshipit-source-id: 075eb40e6f76d4f3271fdf243a5728322698ff46
Summary: looks like we got land time conflict
Reviewed By: krallin
Differential Revision: D24196362
fbshipit-source-id: 27da83a2f86cc7fe5f59fe583d4b719f69df0248
Summary:
We've run into an high cpu usage issue on commit_file_diffs request.
Looks like the problem is in the fact that ChangesetPathContext does fsnode
traversal for each path, which is very expensive if we have a lot of paths
Note - this is similar problem to D20766465 (2eebab89c5)
Reviewed By: mitrandir77
Differential Revision: D24194056
fbshipit-source-id: e808ff7c63990260c1eb2f70a8bba11c2add395c
Summary:
Mononoke command for running the SegmentedChangelogSeeder for an existing
repository. The result is going to be a new IdMap version in the metadata
store and a new IdDag stored in the the blobstore resulting in a brand new
SegmentedChangelog bundle.
Reviewed By: krallin
Differential Revision: D24096963
fbshipit-source-id: 1eaf78392d66542d9674a99ad0a741f24bc2cb1b
Summary:
The SegmentedChangelogSeeder has the role of constructing a new IdMap for a
given repository. That would happen when a repository is onboarded or when
algorithm improvements are made.
This change comes with small refactoring. We had the Dag which did a bit of
everything. Now the on_demand_update and the seeder functionalities are in
their separate files. The tests from `dag.rs` were moved to the `tests.rs` and
updated to use the seeder and on_demand_update structures.
`SegmentedChangelogSeeder::run` is the main logic added in this diff.
Reviewed By: quark-zju
Differential Revision: D24096965
fbshipit-source-id: 0f655e8c226ca0051f3e925342e92b1e7979aab2
Summary:
The IdDagStore provides the ability to save and later load prebuilt instances
of the IdDag.
This is going to be used in the clone API where we send one of these blobs to
the client. It is also going to be used by servers starting up.
Right now the serialization is naive, relying on serde::Serialize. The key
schema would provide the means for evolving the serialization format in cases
where we would require breaking changes.
Reviewed By: quark-zju
Differential Revision: D24096967
fbshipit-source-id: 2c883e5e82c05bec03c429c3c2a2d545170a8c05
Summary:
This IdMapVersionStore determines which is the latest IdMapVersion that commit
"tailing" processes should use when building new Dag bundles. The "seed"
process will update the versions of the IdMap. The plan for the "seed" process
is to write a new IdMap version to Sql then update the store with a new entry.
New "tailer" processes will then start to use the newly built IdMapVersion.
The tailing processes that will build fresh IdDags for general consumption.
These IdDags will be used by the clone operation. These dags will also be used
by servers instances spinning up.
DagBundles specify (id dag version, id map vession). This pair specified a
prebuilt Segmented Changelog that is ready to be loaded.
Reviewed By: quark-zju
Differential Revision: D24096968
fbshipit-source-id: 413f49ed185a770a73afd17dfbc952901ab53b42
Summary:
This allows for more flexibility in structuring the code that wants to read all
the public changesets.
The usecase I have in mind is the SegmentedChangelog Seeder. The logic is
defined in the segmented_changelog crate. Constructing the Seeder is more
straight forward if it doesn't have to take direct dependency on SqlPhases and
SqlChangesets.
Reviewed By: quark-zju
Differential Revision: D24096966
fbshipit-source-id: dffa909cd27d6c05d745fd0fe0609114a50f1892
Summary:
Adding a simple from implementation to the error struct allows us to avoid
instantiating the varint explicitly.
Reviewed By: krallin
Differential Revision: D24161695
fbshipit-source-id: cb6e4c1e2cb21bd17ddff0df89a53d3f0604f562
Summary: Let's use more references when we can
Reviewed By: krallin
Differential Revision: D24161694
fbshipit-source-id: 5cf7edf843fa8dcf0d24ca114c3d520263d92a3b
Summary:
This change enables the filler job to work on all repos available instead of a
single one. We are still going to be able to dedicate the job to a certain repo
(by crafting a config with a single repo enabled) but we can put the entire
long tail for low-traffic repos under a single job.
This requires D24110335 to land in configerator to work.
Reviewed By: krallin
Differential Revision: D24136239
fbshipit-source-id: 4b77d1667c37cc55f11c3087b02a09dbae29db0f
Summary: Allow bookmark to be optional - again, will be used in the next diffs
Reviewed By: ahornby
Differential Revision: D24163608
fbshipit-source-id: e037731117181d0b1bbe4eb273301245142b507d
Summary: This functionality will be used in the next diffs.
Reviewed By: ahornby
Differential Revision: D24163517
fbshipit-source-id: 36e5c9646e21913f0e0d79d77dd11862f5aa5331
Summary:
This diff fixes how syncing of merge commits decides on the `CommitSyncConfigVersion` to use. Old and incorrect behavior just always uses current version from `LiveCommitSyncConfig`. The desired behavior is to reuse the version with which parent commits are synced, and manually sync commits when version changes are needed.
For merges it is more interesting, as merges have multiple parents. The overarching idea is to force all of the parents to have the same version and bail a merge if this is not the case. However, that is an ideal, and we are not there yet, because:
- there are `NotSyncCandidate` parents, which can (and should at the moment) be safely excluded from the list of parents of the synced commit.
- there are `Preserved` parents (which will turn into the ones synced with a `noop` version)
- there are `RewrittenAs` and `EquivalentWorkingCopy` parents, which don't have an associated version.
So until the problems above are solved:
- absent `RewrittenAs`/`EquivalentWorkingCopy` versions are replaced with the current version
- `Preserved` merge parents cause merge sync to fail.
Reviewed By: StanislavGlebik
Differential Revision: D24033905
fbshipit-source-id: c1c98b3e7097513af980b5a9f00cc62d248fc03b
Summary:
Our higher-level goal is to get rid of `CommitSyncOutcome::Preserved` altogether. This diff is a step in that direction. Specifically, this diff removes the creation of "accidental" Preserved commits: the ones where the hashes are identical, although a `Mover` of some version have been applied. There are a few sides to this fix:
- `get_commit_sync_outcome` now returns `Preserved` only when the source and target hashes are identical, plus stored version is `None` (previously it would only look at hashes).
- `sync_commit_no_parents` now records the `Mover` version it used to rewrite the commit (previously it did not, which would sometimes create `Preserved` roots)
- there are now just two ways to sync commits as `Preserved`:
- `unsafe_preserve_commit` (when the caller explicitly asks for it). The idea is to only remove it once we remove the callers of this methods, of course.
- `sync_commit_single_parent` when the parent is also `Preserved`. Note that automatically upgrading from `Preserved` parent to a rewritten changeset is incorrect for now: `Preserved` does not have an associated version by definition, so we would have to use a current version, which may corrupt the repo. Once we get rid of `Preserved`, this case will naturally go away.
- as we now have `update_mapping_with_version` and `update_mapping` (which consumes current version), we need to add explicit `update_mapping_no_version` for preserved commits we are still creating (again, recording a current version is a mistake here, same reason as above)
NB: I've added/changed a bunch of `println`s in tests, leaving them here, as they are genuinely useful IMO and not harmful.
Reviewed By: StanislavGlebik
Differential Revision: D24142837
fbshipit-source-id: 2153d3c5cc406b3410eadbdfca370f79d01471f9
Summary:
Implement BlobstorePutOps for S3Blob. This uses is_present to check the various put behaviours
While implementing this I noticed get_sharded_key could be updated to take a reference, so I did that as well.
Differential Revision: D24079253
fbshipit-source-id: 16e194076dbdb4da8a7a9b779e0bd5fb60f550a6
Summary: Now that fileblob and memblob support put behaviour logic, update the overwrite test to check the overwrite result.
Differential Revision: D24021167
fbshipit-source-id: d9578630205cf5d79999a459cc29481968d5717d
Summary: Update memblob to be PutBehaviour aware by changing implementation from Blobstore to BlobstoreOps
Differential Revision: D24021166
fbshipit-source-id: 04dd25c5535769ea507120c1886592b808a7bbc6
Summary: Update Memblob::new callsites to ::default() in preparation for adding arguments to ::new() to specify the put behaviour desired
Differential Revision: D24021173
fbshipit-source-id: 07bf4e6c576ba85c9fa0374d5aac57a533132448
Summary: Add put behaviour handling to fileblob so that it can prevent overwrites if requested.
Differential Revision: D23933228
fbshipit-source-id: 8e74ac96b232be841174f6ad2bd2fccf92aaa90d
Summary:
Add put behaviour to BlobstoreOptions in preparation for passing in the put behaviour through blobstore_factory.
Later in the stack a command line option is added to set this non-None so that we can turn on overwrite logging for particular jobs.
Reviewed By: StanislavGlebik
Differential Revision: D24021169
fbshipit-source-id: 5692e2d3912ebde07b0d7bcce54b79df188a9f16
Summary: Add a new "user" column to the EdenAPI server's Scuba data.
Reviewed By: krallin
Differential Revision: D24153765
fbshipit-source-id: 95a3177d9283e5b0e3f47e7f42a1de5142049b99
Summary: Add a method to get the client's username from a client identity. This is helpful for logging, etc.
Reviewed By: krallin
Differential Revision: D24153766
fbshipit-source-id: 2ecf85e5de72918aeb292ce88539a991da4de900
Summary: Allow source control service clients to set pushvars.
Reviewed By: krallin
Differential Revision: D24136870
fbshipit-source-id: 34f9176ec66ca319b363c91015dae9b59a55a244
Summary:
Add the possibilty of setting `pushvars` when deleting bookmarks. This makes
it consistent with the other write operations.
Reviewed By: krallin
Differential Revision: D24136869
fbshipit-source-id: f98b74c6c731e50552184000ad697d04748711fd
Summary:
Previously all mutable_counters reads were going to leader. This might be
useful for some cases, but in the case of blobimport warmer this opens too
many connections to sql leader.
Let's read from replica instead
Reviewed By: krallin
Differential Revision: D24160315
fbshipit-source-id: 0cecde3c54a00bbea215a1e0fa63d4a7c3dc9eaa
