Summary:
Extract the cache of warmed-up bookmarks (those for which derived data has been
computed) from the apiserver code so that we can re-use it.
Reviewed By: krallin
Differential Revision: D17913440
fbshipit-source-id: 97bf980d4c9dc3fff93000e3d30f6ea3b783bd43
Summary: Print calculated statistics in csv format, so we can generate csv file by redirecting output to some file. As a TODO we may want to add new fields to RepoStatistics struct, refactor code and create csv file using e.g. serde deserialization.
Reviewed By: krallin
Differential Revision: D17907650
fbshipit-source-id: 0e7f0af522cc72c067d59431039e44998d5dd354
Summary:
Changes to blobrepo to make it more reusable from graph_walker
* Add open method that recieves underlying blobstore. Planning to use this from graph_walker.
* Only wait for myrouter in xdb mode.
Reviewed By: krallin
Differential Revision: D17113943
fbshipit-source-id: 6794c2e1b8ec7ff1ebba2ecdf7c5cc7963ca7b32
Summary: They are always used together, so let's just bundle them.
Reviewed By: krallin
Differential Revision: D17910811
fbshipit-source-id: 21afc43814d9fd513fc02498e7dae1e68606e389
Summary: Removes the dependency of ////common/rust/thrift/runtime:fbthrift// on ////common/rust/bytes-ext//. This is a move toward a self-contained runtime that is easy to open source.
Reviewed By: yfeldblum
Differential Revision: D17891341
fbshipit-source-id: ddc53735c3ecde32e16a10bf98ae24a68aec9d82
Summary: The BufExt trait only exists for Thrift. This change moves it into our fbthrift runtime which already contains a similar BufMutExt. This is a move toward a self-contained runtime that is easy to open source.
Reviewed By: yfeldblum
Differential Revision: D17891338
fbshipit-source-id: 17fe9d672ebb866e47c47bbd7c3b7c3da8d327ef
Summary:
Some backfill types, e.g. fsnodes, require access to the unredacted repo in
order to correctly compute their values. For example, fsnodes require the
content hashes of the redacted blobs to correctly form the fnsode graph.
This kind of access is ok, as we are only deriving summary information
(hashes), and won't reveal the content of the redacted blob.
Add the ability for `backfill_derive_data` to access the unredacted repo,
limited only to whitelisted derived data types.
Reviewed By: ikostia
Differential Revision: D17877191
fbshipit-source-id: 6c9b1dfdfb2e6f431815ddf3de60029fbb180454
Summary: This lets a user of a megarepo look up subrepo hashes
Reviewed By: ikostia
Differential Revision: D17831869
fbshipit-source-id: 16962fd7b09053e4a7be196ccd0f52c700626bdd
Summary:
Follow up from D17765160. Let's unify logview initialization in one place.
This will let us log to logview from all binaries we have in mononoke
Note - the diff touches a lot of files, but main changes are in
cmdlib/src/args.rs, apiserver/src/main.rs and server/src/main.rs.
Reviewed By: krallin
Differential Revision: D17895480
fbshipit-source-id: c922adfb385461ff168bd788e42ea1b88891f7cf
Summary: I'll add WireprotoLogging struct, so need to rename this one
Reviewed By: ikostia
Differential Revision: D17878970
fbshipit-source-id: 2966f5cb8c8d2399e1691ef1c710dd9362a976ee
Summary:
This diff updates all license headers to use the new text and style.
Also, a few internal files were missing the header, but now they have it.
`fbcode/common/rust/netstring/` had the internal header, but now it has
GPLV2PLUS - since that goes to Mononoke's Github too.
Differential Revision: D17881539
fbshipit-source-id: b70d2ee41d2019fc7c2fe458627f0f7c01978186
Summary:
Similarly to `Mover`, which acts on paths according to the sync config,
we need a `BookmarkRenamer`, which acts on bookmark names.
The logic now supports two types of bookmark configs:
- common pushrebase bookmarks: they are common for all small repos and they
don't get renamed. `master` is one example of those, and I don't know if
there can be any other :)
- prefixes: when the bookmark is updated in the small repo, an equivalent,
but prefixed bookmark is updated in the large repo. Correspondingly, when
a prefixed bookmark is updated in a large repo, an equivalent, but unprefixed
bookmark is updated in the small repo.
There are also "error" cases:
- bookmark is neither common pushrebase, nor prefixed with the correct small
repo's prefix. In that case we plan not to update that bookmark (although,
create the corresponding commit(we may have to be careful and make it draft
instead of public or something like this))
Reviewed By: krallin
Differential Revision: D17877749
fbshipit-source-id: 6e2f67456de354c2358376e5762fabffe0de3a42
Summary: Added option to calculate statistics for old changesets saved in file. Currently it prints calculated statistics.
Reviewed By: StanislavGlebik
Differential Revision: D17811305
fbshipit-source-id: 162946941b9e153ffedc1fc28539eebacab77132
Summary:
Now that we don't fire this error from bundle2_resolver, let's also not
create it from there.
Reviewed By: StanislavGlebik
Differential Revision: D17856088
fbshipit-source-id: 4e0ec896dcb4047c01bc9d858927d6e432b43962
Summary:
See bottom diff of the stack for the entire stack's purpose.
The goal of this particular diff is to address step 2:
Things that are only used by `repo_client::unbundle` and not by `bundle2_resolver`, should be moved out of `bundle2_resolver`. While moving such things, it will turn out that some things that `Bundle2Resolver` struct owned (and some things that `bundle2_resolver::resolver::resolve` function accepted as arguments) are only threaded down to whichever post-resolve action is appropriate. Such things should just not be passed into `resolve` in the first place.
In theory, after this diff it is already possible to introduce something in between the `resolve` and `run_post_resolve_action` calls.
Reviewed By: StanislavGlebik
Differential Revision: D17853603
fbshipit-source-id: 0442b478dad82723383896666096eec2aec64e58
Summary:
The goal of this stack is:
- extract the actual "business logic" of `unbundle` wireproto command out of `bundle2_resolver`
- make `bundle2_resolver` only parse the bundle, save uploaded blobs/changesets and return the desired next action with the required arguments
- run the action (`PostResolveAction`) in the `repo_client::RepoClient::unbundle`
This allows us to later insert the push redirection logic into the `unbundle()` fn like this:
1. `bundle2_resolver` resolves the uploaded bundle and saves all the changesets. At this point, instead of dealing with the stream of bytes we deal with uploaded changesets and a full knowledge of what needs to happen next (push, infinitepush, pushrebase or bookmark-only pushrebase)
1. check if push redirection needs to happen. If it does not, just to the last step.
1. Run commit sync on all of the uploaded changesets, thus creating them in the target repo. Then replace source repo commit ids with target repo commit ids in action arguments
1. Run the action in the target repo
1. Wait until the created changesets are synced into the source repo, generate and record (hg) changeset ids in the source repo
1. Construct a correct reply part
This is a cumbersome refactoring, therefore to make it easier to review (and write and think about) I break it down into multiple steps:
1. (this diff) Introduce the `PostResolveAction` `enum` and make sure `bundle2_resolver::resolve` returns it instead of a created response. Make `repo_client::RepoClient::unbundle` look at this `enum` and run the appropriate action from the newly-created `repo_client::unbundle` module. No business logic change, everything is completely mechanical, just copy-pastes. Because I don't yet move the functions/structs from `bundle2_resolver::resolver`, I need to make some things `pub` that I wouldn't otherwise. The goal is to move them into `unbundle` later and remove `pub` as I go. The goal is also to eventually never leak `BundleResolver` struct out of `bundle2_resolver::resolver` file.
1. Things that are only used by `repo_client::unbundle` and not by `bundle2_resolver`, should be moved out of `bundle2_resolver`. While moving such things, it will turn out that some things that `Bundle2Resolver` struct owned (and some things that `bundle2_resolver::resolver::resolve` function accepted as arguments) are only threaded down to whichever post-resolve action is appropriate. Such things should just not be passed into `resolve` in the
first place.
1. Separate `Push` and `InfinitePush` processing (simultaneously introducing a new variant to `PostResolveAction`)
Reviewed By: StanislavGlebik
Differential Revision: D17831366
fbshipit-source-id: f19f9d7a7abed28d1c6bf519ef81942699b892bc
Summary:
Previously function remapped each copy_from changeset separately, however this
is not necessary. copy_from changeset is guaranteed to be one of the parents
(see https://fburl.com/0i80ojja).
This diff simplifies the function a bit by not doing a separate remap for
copy_from changesets
Reviewed By: farnz
Differential Revision: D17779792
fbshipit-source-id: c4c64bdc295858289a26e1d5b229ced6eca96414
Summary:
So far we did not really care how uploaded hg changesets map into bonsais. This
changes however, as we start thinking about push redirection. In these
scenarios we need to upload hg changesets in repos for which they are destined,
then commit-sync their bonsai equivalents to redirection targets, run
push/pushrebase logic in these target repos and await back-sync.
As part of this scenario, we need bonsais for:
- syncing them into target repos
- using their ids for any dependent logic (e.g. push needs to know the
changeset id to which bookmark needs to point)
Note that this diff does not really change any real behavior, just preserves
uploaded bonsais for a bit. Thus running existing test suite should be enough
to verify it.
Reviewed By: StanislavGlebik
Differential Revision: D17811012
fbshipit-source-id: 0f042f5c765a0f961c332f91dfe8820aa4929193
Summary:
This updates the blobstore healer to avoid using a hardcoded list of regions.
Facebook
I'll remove this from the TW spec once this makes it through Conveyor.
This Configerator file is referenced in the DBA region turnup instructions: https://our.intern.facebook.com/intern/wiki/MySQL/DBaaS/Internal/Regional_Turnup_Steps/#configs (they update `configerator/raw_configs/myrouter/routing_v3.jsonc`, which includes this file)
Reviewed By: StanislavGlebik
Differential Revision: D17787579
fbshipit-source-id: ec9128202679ce0dbe18d77fb84c3fbf6186cd19
Summary:
There's a few things broken with common/rust/sql and the blobstore healer's handling of replication lag right now:
- If Seconds_Behind_Master isn't an int (it'll be NULL if replications is paused), it just panics.
- If it's talking to a server that it expected to be a replica but is a master, it returns None for the replication lag, but 0 would be more appropriate.
- If a region no longer has a replica, it errors out.
This diff fixes that:
- If replication is paused, we return None for lag.
- If we're talking to a master, we return 0.
- If a region has no replica, we ignore it.
Reviewed By: StanislavGlebik
Differential Revision: D17787580
fbshipit-source-id: 9e5e7682456870b88910afec12e1c409fd8c5ba6
Summary:
This updates cache warmup to avoid using `recursive_entry_stream`, and to
instead use `ManifestOps::list_all_entries`. Notably, the
`recursive_entry_stream` implementation tends to get slower the more things we
add (P116031891), whereas the bounded traversal-based implementation does not
(P116031745), which makes the latter faster.
Reviewed By: StanislavGlebik
Differential Revision: D17763152
fbshipit-source-id: e5df452472f4dcd90bbdfa1523d01afbd8189c30
Summary:
This is better from the usability perspective, as it can resolve hg changeset, bonsai changeset and a
bookmark.
However, this also means that when the user does provide an Hg commit hash and we need an hg commit hash, we are doing one more database query (`get_hg_from_bonsai_changeset`). This is a little bad, but all of the use-cases I changed here are from the admin tool, so we should be fine.
Reviewed By: StanislavGlebik
Differential Revision: D17740448
fbshipit-source-id: 7c1620979e25631f1a4e44d6310668fe634b2075
Summary: It should be used only in pushrebase replay tests
Reviewed By: krallin
Differential Revision: D17746350
fbshipit-source-id: 5c0877b4b554ad53f0fe3228a696c5730dc6958f
Summary:
This udpates the apiserver to spawn requests on a Tokio runtime, instead of an
Actix arbiter (which does contain a Tokio runtime, but it's a `current_thread`
one).
This allows us to make use of more than one core when serving Thrift requests.
Reviewed By: HarveyHunt
Differential Revision: D17738537
fbshipit-source-id: 5d17fa73e0185342dc6976de1423919ba545ca79
Summary:
This silences logging from various cpp libraries we use (but not our own
logging) in tests. This should remove some flakyness in our tests coming from
those.
Reviewed By: StanislavGlebik
Differential Revision: D17475237
fbshipit-source-id: ecee69b543d1b431d1da883f67fbc30915697e13
Summary: Implmement the `file_content_chunk` method, which gets a chunk of file content.
Reviewed By: StanislavGlebik
Differential Revision: D17686286
fbshipit-source-id: 0330aef9f0be1dea3040e9ba3e767867165827e7
Summary:
Add methods to fetch file contents, either for the full file, or part of the
file.
Reviewed By: StanislavGlebik
Differential Revision: D17686294
fbshipit-source-id: 8bceee63db13e5eac274f8372631e554a6583036
Summary:
Allow clients to fetch parts of a file specified by range, rather than the
whole file.
For chunked files the fetch skips over chunks which don't overlap with the
range, so only relevant chunks are fetched from the blobstore.
Reviewed By: krallin
Differential Revision: D17672378
fbshipit-source-id: b1cd2f6b9f069ce86f7d1bccd610f4ff9d767f3e
Summary: Skip calculating number of lines for files with size greater than 10MB
Reviewed By: krallin
Differential Revision: D17710287
fbshipit-source-id: 025e94b66539b471e5e48aee0ec7e61808cb0bd6
Summary:
Attached tasks have more details, but in short - if a commit has two parents
then the order of them in mercurial can change but the commit hash won't
change. In Mononoke it's different - change of parent order changes the hash.
This option allows us to fix parent order if needed
Reviewed By: krallin
Differential Revision: D17683807
fbshipit-source-id: 23549e55369b625ea5d597bc336b465373cbbc74
Summary:
We'd like to log hook names so that it's easier to add monitor and alarm if a
single hook is failing.
Note that I had to make a small fix on stats side.
Reviewed By: farnz
Differential Revision: D17686683
fbshipit-source-id: 06d3d72477ecdfe02df34633fe100be64c15f2d0
Summary:
I ran into one more problem with generating pushrebase bundles - if a bookmark was
deleted, then recreated again then sync job might send commits that already
exists on hg server and that causes sync failure.
There's a way to make it work (tracking mercurial heads instead of bookmarks),
however at this point I feel like it's not worth it - generating normal push
bundles should be easier.
Reviewed By: farnz
Differential Revision: D17683425
fbshipit-source-id: a2a7d2bf6a65a9a85de7de0c44cd12eb23e01727
Summary: print_statistics and log_statistics were used in the same places in code.
Reviewed By: StanislavGlebik
Differential Revision: D17665790
fbshipit-source-id: 5d41f1b9d77edbe12db25f109a8bb171c8072fb4
Summary:
UploadEntries::finalize() function is called during pushes and blobimports i.e.
when new commit is created from hg changeset. It does a few checks, one of them
is making sure that all entries referenced by a manifest are present in the
blobstore.
The problem is that it's actually quite wasteful - if a single file was
modified in a directory then it'll check that all of the entries in this
directory are present.
Let's change the logic and check that only entries that were added by this
commit are present in the blobstore (this is what find_intersection_of_diffs is
doing). That doesn't make it less correct - if an entry is referenced in a
manifest but not added in this commit then it will be checked in one of the
parent commits.
Reviewed By: farnz
Differential Revision: D17664062
fbshipit-source-id: 6e7e16084c9126bdb757793d441707fada5052ff
Summary: Set time when changeset was created instead of current time (default time) in samples that are saved in Scuba
Reviewed By: StanislavGlebik
Differential Revision: D17629069
fbshipit-source-id: a4c34f38eff520b96f7e8a85445ee2255c220941
