Summary:
1) Turned out it's possible to have non-prefix free paths in aosp manifests. So
we have to remove this check for now
2) also let's verify config earlier so that we can return an error to the user
faster
Differential Revision: D29335602
fbshipit-source-id: 3dd72d63a370515eca5d356b3b98bb2ac2245aee
Summary: There is a regression in 1.7.0 (which we're on at the moment) so we might as well update.
Reviewed By: zertosh, farnz
Differential Revision: D29358047
fbshipit-source-id: 226393d79c165455d27f7a09b14b40c6a30d96d3
Summary:
Path should be relative to the symlink path, not to the repo root. This diff
fixes it
Reviewed By: farnz
Differential Revision: D29327682
fbshipit-source-id: a51161a8039a88263fe941562f2c2134aa5d4fef
Summary:
Pull in a patch which fixes writing out an incorrect entsize for the
`SHT_GNU_versym` section:
ddbae72082
Reviewed By: igorsugak
Differential Revision: D29248208
fbshipit-source-id: 90bbaa179df79e817e3eaa846ecfef5c1236073a
Summary: Update versions for several of the crates we depend on.
Reviewed By: danobi
Differential Revision: D29165283
fbshipit-source-id: baaa9fa106b7dad000f93d2eefa95867ac46e5a1
Summary: revert the zstd crates back to previous version
Reviewed By: johansglock
Differential Revision: D29038514
fbshipit-source-id: 3cbc31203052034bca428441d5514557311b86ae
Summary:
Turned out we still don't process copyfile attribute correctly. copyfile
creates two overrides like
{
"copyfile source" -> default prefix + "/copyfile source"
"copyfile source" -> "destination"
}
However we also have a check that validates that there no paths (that includes
default path, overrides, linkfiles etc) that are prefixes of each other. And
this check fails in the copyfile case, even though the path maps to exactly the
same path as if default prefix was applied.
Let's fix it. Also note that we haven't seen this failure in the integration
test because we didn't run verify_paths in tests. This diff fixes it as well.
Reviewed By: mitrandir77
Differential Revision: D28992456
fbshipit-source-id: 5fd993914b189cf768ba03010194b1c26026f7a8
Summary: Update to latest version. This includes a patch to async-compression crate from [my PR updating it](https://github.com/Nemo157/async-compression/pull/125), I will remove once the crate is released.
Reviewed By: mitrandir77
Differential Revision: D28897019
fbshipit-source-id: 07c72f2880e7f8b85097837d084178c6625e77be
Summary: I need it for test in next diff
Reviewed By: StanislavGlebik
Differential Revision: D28959433
fbshipit-source-id: c67ca7eec03f94425332e446f6f97038edff598d
Summary:
Summar:
I'll be using it more in the next diff, so why not have it in it's own method.
Reviewed By: StanislavGlebik
Differential Revision: D28887333
fbshipit-source-id: 35accb495a577e1c01ec8114fc60acf38ed11fee
Summary:
When syncing merge commits with two parents it would be nice if it was the first parent that comes from the unified branch. In **case of octopus merges** we really don't want
the parent in unified branch to be third (that would turn the sync into
non-forward move!). Let's add a way to tell the commit rewriter which parent
needs to be first.
Reviewed By: farnz
Differential Revision: D28885488
fbshipit-source-id: 57a081ce2d285ba2b6d6d98110cd1c64a241548e
Summary:
The worker should be able to process the requests from the queue no matter
which repo it is and what are it ACLs. It's during the request scheduling when
we should check the identity of the entity scheduling request.
Reviewed By: StanislavGlebik
Differential Revision: D28866807
fbshipit-source-id: 5d57eb9ba86e10d477be5cfc51dfb8f62ea16b9e
Summary: This makes it easier to understand what each move and merge commits are for.
Reviewed By: mitrandir77
Differential Revision: D28839677
fbshipit-source-id: 1a42205c164224b64c773cff80b690b251a48381
Summary:
This is a precaution. add_sync_target can create a very branchy repository, and
I'm not sure how well Mononoke is going to handle deriving of these commits by
all mononoke hosts at once (in particular, I'm worried about traversal that has
to be done by all hosts in parallel). In theory it should work fine, but
deriving data during add_sync_target call should be a reasonable thing to do
anyway.
While working on that I noticed that "git" derived data is not supported by derived data utils, and it was causing test failures. I don't think there's any reason to not have TreeHandle in derived data utils, so I'm adding it now.
Reviewed By: farnz
Differential Revision: D28831052
fbshipit-source-id: 5f60ac5b93d2c38b4afa0f725c6908edc8b98b18
Summary:
source_name to be unique even if the same git-repo and project name is mapped several times in the same manifest.
source_name need to match between all the different places we use it, `add_target_config, sync_changeset, changesets_to_merge` we where not consistent.
Reviewed By: StanislavGlebik
Differential Revision: D28845869
fbshipit-source-id: 54e96dcdeaf22ec68f626e9c30e5e60c54ec149b
Summary:
We are going to create quite a lot of move commits at the same time, and it can
be slow. Let's instead create them in parallel, and then call
`save_bonsai_changesets` for all the commits in one go.
Reviewed By: mitrandir77
Differential Revision: D28795525
fbshipit-source-id: f6b6420c2fe30bb98680ac7e25412c55c99883e0
Summary:
Previously add sync target method was creating a single merge commit. That
means that we might create a bonsai commit with hundreds of parents. This is
not ideal, because mercurial can only work correctly with 2 parents - for
bonsai changeset with 3 parents or more mercurial file histories might be lost.
So instead of creating a single giant merge commit let's create a stack of
merge commits.
Reviewed By: mitrandir77
Differential Revision: D28792581
fbshipit-source-id: 2f8ff6b49db29c4692b7385f1d1ab57986075d57
Summary:
This fixes the potential race condition in megarepo methods where the bookmark
is updated between executing the actual method and returning the current
bookmark value
Reviewed By: StanislavGlebik
Differential Revision: D28796099
fbshipit-source-id: 472f9bbcfafe1c2eb78c62ccf88d149b5157b646
Summary:
Until now we had no way of distinguishing the following two situations:
* request failed and the returned error represents this failure
* pooling request failed, the state of actual request is unknown (it might not have even finish yet!)
This diff changes the returned structure to provide the union of error and
response. This has also an upside of simplifying code as the thrift result is
**exactly** what we store in our blobstore.
Reviewed By: StanislavGlebik
Differential Revision: D28747624
fbshipit-source-id: 60473d1491a8c8a016838ec9b2fea0939af10079
Summary:
For some reason the async requests errors were stored in blobstore as strings.
We already have datastructures to properly store them without information loss:
the actual thrfit exceptions that are going the be returned to the end user, we
can't use them directly but we can create structs that with the same structure
so the casting between them will be trivial.
This diff changes what we store. Unfortunately we still deserialize to
MononokeError and serialize back to thrift on pooling. The rest of this diff
stack will deal with that.
Reviewed By: StanislavGlebik
Differential Revision: D28747623
fbshipit-source-id: 7506f1a29220b49482f40d1db8e6b552dabd679a
Summary: Tests take too much space - let's move it to a separate file.
Reviewed By: mitrandir77
Differential Revision: D28712730
fbshipit-source-id: 904afc965b43b21438cb939260967a6057939c59
Summary:
This diff adds a verification that checks that there are no two source files that
remap to the same file in target.
Reviewed By: mitrandir77
Differential Revision: D28709871
fbshipit-source-id: 9b407d223d1d431f2a28031c70b982630c97bdbf
Summary:
Linkfiles are supported by grepo tool - this is a symlink to a file in a sub
git repo.
Previously add_sync_target ignored them - let's fix it now.
Reviewed By: mitrandir77
Differential Revision: D28704032
fbshipit-source-id: 49e66200c0ce8c46d6fb7d81d5f948a5842a064c
Summary:
An operation that takes a result of computations and saves it so it can be
polled.
Reviewed By: krallin
Differential Revision: D28606343
fbshipit-source-id: 4fdd48f40812c55842669ea3f62865fc8244f6cc
Summary:
This abstracts the constant poolling for new items that megarepo workers will
be doing.
Reviewed By: krallin
Differential Revision: D28606344
fbshipit-source-id: 159c78c207956bcf893ffb68f7f7e93fc2426c8b
Summary: a dequeue operation abstracts popping a request from the queue for processing
Reviewed By: krallin
Differential Revision: D28606339
fbshipit-source-id: d8eba568eab7b6751e422cd11e8ffa5bcb7712b0
Summary: This diff switches megarepo_api to use the types created in the previous diffs
Reviewed By: krallin
Differential Revision: D28606338
fbshipit-source-id: c048f274395239528b3a0280172dd21052055111
Summary:
Instead of storing params separate types for each type of request let's store
them in a single union type. This is going to simplify things and improve the
type safety (as we will no longer need to trust in the "request_type" string
store in sql to safely deserialize the type).
Reviewed By: krallin
Differential Revision: D28606341
fbshipit-source-id: cfa8ee15213e6582891906e326d0e258c3a918a1
Summary: This operation is primary way of getting new jobs from queue to tailer.
Reviewed By: krallin
Differential Revision: D28441669
fbshipit-source-id: e4563d74d537d826ad4eba654fe583ea55aee974
Summary: We do the same thing in many places which isn't great for code readability.
Reviewed By: krallin
Differential Revision: D28441674
fbshipit-source-id: bec69d3bc8a000ac9442020f5054e5c79b3392de
Summary:
This is the very first version of add_sync_target method. It's very naive - it
just creates a single merge commit, and it doesn't handle all the necessary
functionality (e.g. linkfiles) and it doesn't do all the necessary checks yet.
TODOs show the missing functionality.
However this implementation it should be good enough as the first version, and
we can continue to extend it.
Reviewed By: mitrandir77
Differential Revision: D28677365
fbshipit-source-id: a8af6b92ced285ce4d012843f591d5c29584669d
Summary: It will be used in the next diffs, let's move it to a common file
Reviewed By: mitrandir77
Differential Revision: D28675768
fbshipit-source-id: 78c21e378053a157d09ee9d032576a434a52c912
Summary: They will be used in the next diffs
Reviewed By: mitrandir77
Differential Revision: D28675756
fbshipit-source-id: 2f41191cb83921fb3ff3374c0ccb7a884b7b3a25
Summary:
For debugging and better error messages it would be nice to know who worked on
a given request.
I'll do AOSC schema change once accepted.
Reviewed By: krallin
Differential Revision: D28441673
fbshipit-source-id: ba146d7f43dde26d9433f76af7fe982da14b5b82
Summary:
Those will be used for two purposes:
* to limit a scope of given tailer to just given repo (this way we could use
different tailer binaries for different repos or disable processing for a
single repo etc...)
* to enforce a single in-flight request per repo (to prevent client from
accidentally scheduling duplicate requests etc...)
I'll do AOSC schema change once accepted.
Reviewed By: krallin
Differential Revision: D28441670
fbshipit-source-id: 7b35a1c7034707d7cf54220c559edd6e03f430c3
Summary: I want to put more things in the async_requests crate.
Reviewed By: krallin
Differential Revision: D28441671
fbshipit-source-id: 19233c2c5b697cc1e27107cd9904666baf8f10b7
Summary:
Simple test that syncs a single commit from a source to a target.
A few notes
1) I want to use test Mononoke instance, but I can't initialize them in another
crate because of cfg(test). I removed cfg(test) to make it possible.
2) I initialized megarepo mapping in TestRepoFactory so that all tables
were in the same db. It's not strictly necessary, but makes initialization a bit simpler.
Reviewed By: mitrandir77
Differential Revision: D28444865
fbshipit-source-id: f39c917806709606ce8e5a1c1158f186d239d8b8
