Summary:
This updates the blobstore healer to avoid using a hardcoded list of regions.
Facebook
I'll remove this from the TW spec once this makes it through Conveyor.
This Configerator file is referenced in the DBA region turnup instructions: https://our.intern.facebook.com/intern/wiki/MySQL/DBaaS/Internal/Regional_Turnup_Steps/#configs (they update `configerator/raw_configs/myrouter/routing_v3.jsonc`, which includes this file)
Reviewed By: StanislavGlebik
Differential Revision: D17787579
fbshipit-source-id: ec9128202679ce0dbe18d77fb84c3fbf6186cd19
Summary:
There's a few things broken with common/rust/sql and the blobstore healer's handling of replication lag right now:
- If Seconds_Behind_Master isn't an int (it'll be NULL if replications is paused), it just panics.
- If it's talking to a server that it expected to be a replica but is a master, it returns None for the replication lag, but 0 would be more appropriate.
- If a region no longer has a replica, it errors out.
This diff fixes that:
- If replication is paused, we return None for lag.
- If we're talking to a master, we return 0.
- If a region has no replica, we ignore it.
Reviewed By: StanislavGlebik
Differential Revision: D17787580
fbshipit-source-id: 9e5e7682456870b88910afec12e1c409fd8c5ba6
Summary:
This updates cache warmup to avoid using `recursive_entry_stream`, and to
instead use `ManifestOps::list_all_entries`. Notably, the
`recursive_entry_stream` implementation tends to get slower the more things we
add (P116031891), whereas the bounded traversal-based implementation does not
(P116031745), which makes the latter faster.
Reviewed By: StanislavGlebik
Differential Revision: D17763152
fbshipit-source-id: e5df452472f4dcd90bbdfa1523d01afbd8189c30
Summary:
This is better from the usability perspective, as it can resolve hg changeset, bonsai changeset and a
bookmark.
However, this also means that when the user does provide an Hg commit hash and we need an hg commit hash, we are doing one more database query (`get_hg_from_bonsai_changeset`). This is a little bad, but all of the use-cases I changed here are from the admin tool, so we should be fine.
Reviewed By: StanislavGlebik
Differential Revision: D17740448
fbshipit-source-id: 7c1620979e25631f1a4e44d6310668fe634b2075
Summary: It should be used only in pushrebase replay tests
Reviewed By: krallin
Differential Revision: D17746350
fbshipit-source-id: 5c0877b4b554ad53f0fe3228a696c5730dc6958f
Summary:
This udpates the apiserver to spawn requests on a Tokio runtime, instead of an
Actix arbiter (which does contain a Tokio runtime, but it's a `current_thread`
one).
This allows us to make use of more than one core when serving Thrift requests.
Reviewed By: HarveyHunt
Differential Revision: D17738537
fbshipit-source-id: 5d17fa73e0185342dc6976de1423919ba545ca79
Summary:
This silences logging from various cpp libraries we use (but not our own
logging) in tests. This should remove some flakyness in our tests coming from
those.
Reviewed By: StanislavGlebik
Differential Revision: D17475237
fbshipit-source-id: ecee69b543d1b431d1da883f67fbc30915697e13
Summary: Implmement the `file_content_chunk` method, which gets a chunk of file content.
Reviewed By: StanislavGlebik
Differential Revision: D17686286
fbshipit-source-id: 0330aef9f0be1dea3040e9ba3e767867165827e7
Summary:
Add methods to fetch file contents, either for the full file, or part of the
file.
Reviewed By: StanislavGlebik
Differential Revision: D17686294
fbshipit-source-id: 8bceee63db13e5eac274f8372631e554a6583036
Summary:
Allow clients to fetch parts of a file specified by range, rather than the
whole file.
For chunked files the fetch skips over chunks which don't overlap with the
range, so only relevant chunks are fetched from the blobstore.
Reviewed By: krallin
Differential Revision: D17672378
fbshipit-source-id: b1cd2f6b9f069ce86f7d1bccd610f4ff9d767f3e
Summary: Skip calculating number of lines for files with size greater than 10MB
Reviewed By: krallin
Differential Revision: D17710287
fbshipit-source-id: 025e94b66539b471e5e48aee0ec7e61808cb0bd6
Summary:
Attached tasks have more details, but in short - if a commit has two parents
then the order of them in mercurial can change but the commit hash won't
change. In Mononoke it's different - change of parent order changes the hash.
This option allows us to fix parent order if needed
Reviewed By: krallin
Differential Revision: D17683807
fbshipit-source-id: 23549e55369b625ea5d597bc336b465373cbbc74
Summary:
We'd like to log hook names so that it's easier to add monitor and alarm if a
single hook is failing.
Note that I had to make a small fix on stats side.
Reviewed By: farnz
Differential Revision: D17686683
fbshipit-source-id: 06d3d72477ecdfe02df34633fe100be64c15f2d0
Summary:
I ran into one more problem with generating pushrebase bundles - if a bookmark was
deleted, then recreated again then sync job might send commits that already
exists on hg server and that causes sync failure.
There's a way to make it work (tracking mercurial heads instead of bookmarks),
however at this point I feel like it's not worth it - generating normal push
bundles should be easier.
Reviewed By: farnz
Differential Revision: D17683425
fbshipit-source-id: a2a7d2bf6a65a9a85de7de0c44cd12eb23e01727
Summary: print_statistics and log_statistics were used in the same places in code.
Reviewed By: StanislavGlebik
Differential Revision: D17665790
fbshipit-source-id: 5d41f1b9d77edbe12db25f109a8bb171c8072fb4
Summary:
UploadEntries::finalize() function is called during pushes and blobimports i.e.
when new commit is created from hg changeset. It does a few checks, one of them
is making sure that all entries referenced by a manifest are present in the
blobstore.
The problem is that it's actually quite wasteful - if a single file was
modified in a directory then it'll check that all of the entries in this
directory are present.
Let's change the logic and check that only entries that were added by this
commit are present in the blobstore (this is what find_intersection_of_diffs is
doing). That doesn't make it less correct - if an entry is referenced in a
manifest but not added in this commit then it will be checked in one of the
parent commits.
Reviewed By: farnz
Differential Revision: D17664062
fbshipit-source-id: 6e7e16084c9126bdb757793d441707fada5052ff
Summary: Set time when changeset was created instead of current time (default time) in samples that are saved in Scuba
Reviewed By: StanislavGlebik
Differential Revision: D17629069
fbshipit-source-id: a4c34f38eff520b96f7e8a85445ee2255c220941
Summary: This is very important.
Reviewed By: StanislavGlebik
Differential Revision: D17629412
fbshipit-source-id: 36d056b28c7bde324d478f94bac228bd1aff5238
Summary:
In certain circumstances we will fetch a blob from the upstream
rather than internal. Log the oids of blobs that we fetch from upstream.
Reviewed By: farnz
Differential Revision: D17605435
fbshipit-source-id: 23a202dcd57f8af12586449f0d9fc2fd251c7114
Summary:
This way, when Tupperware tells us to terminate, we'll start reporting
unhealthy, which means Proxygen will stop sending traffic to us, and then we'll
exit gracefully a few minutes later.
Reviewed By: HarveyHunt
Differential Revision: D17626009
fbshipit-source-id: 480ae9fdaab97419199d6c05fea51f7c6d6c40f6
Summary: This adds support for gracefully shutting down the LFS Server when SIGINT or SIGTERM is received. This will ensure that we don't close in-flight connections when restarting LFS instances.
Reviewed By: StanislavGlebik
Differential Revision: D17626010
fbshipit-source-id: 32b27e888346cc052a8256a3f7685a9bd60afa46
Summary: Note that it should be possible to replace i64 with usize in a couple of places. That said, it feels like a riskier change than just changing i32 to i64.
Reviewed By: krallin
Differential Revision: D17626031
fbshipit-source-id: 140feb5040054363f7d27bdc4b285e2ff0643f8e
Summary:
scmquery_exporter is a tool to replace svn export and it needs 3 method from scmquery:
- log: to resolve a bookmark to a hash so later calls will be consistent
- ls and cat: to list and download file contents
ScmQuery proxy is currently not accessible from Corp to Prod VIP and will be available soon.
However people using the `js1 upgrade` tool is currently blocked due to the svn deprecation and we
have to make this tool work again ASAP.
The easiest way to make it work, comparing to all other worse approach, is to add this method
in mononoke api server and therefore we can add a backend in scmquery_export.
Discussion: https://fb.workplace.com/groups/JavaScriptOne/permalink/2454194331532724/
Reviewed By: StanislavGlebik
Differential Revision: D17605626
fbshipit-source-id: 19f229123cd980089e1e896c592a0f1afa3c531d
Summary:
There's a corner case that wasn't covered by the new sync job - if a user does
a "hg push --force" which pushes new commits to the server AND moves existing
bookmark then sync job just failed to sync. It tried to do a pushrebase of new
commits on top of existing bookmarks and that would result either in hash
mismatch or a pushrebase conflicts.
To fix it let's generate force pushrebase push in the sync job if bookmark did
non-fast forward move (i.e. "from" is not an ancestor of "to").
Note that there was an option to always do force pushrebase. That should work however that
would make stackpush optimization that we have on hg server impossible and that
might increase sync latency (see https://fburl.com/jpflbd67). Note that it's possible
that we'll switch to use normal push instead of pushrebase in the sync job,
however for now I'd like to go with pushrebase approach because it's exercised much more often.
Reviewed By: krallin
Differential Revision: D17624543
fbshipit-source-id: 1d6121a4767f67fbbbadae712ae8ab9d695857d5
Summary:
After that diff the sync job won't require bundle_replay_data at all if
--generate-bundles flag is specified.
Reviewed By: farnz
Differential Revision: D17605727
fbshipit-source-id: 8fdd43ce7eae41dcf586b17446339c6ba60e960d
Summary:
As suggested by yfeldblum in https://our.intern.facebook.com/intern/diff/D17544447/?transaction_id=2964022370277861.
> the `rust_` bit seems superfluous; the `_thrift` bit opens the door for confusion with apache thrift. Any issue using a module name like `fbthrift`?
---
```
$ fastmod '\brust_thrift\b' fbthrift -d ~/fbcode -e rs,toml
$ fastmod '\brust_thrift\b' fbthrift -d ~/fbcode -g '**/TARGETS'
$ hg revert experimental
$ arc lint -a
```
Reviewed By: bolinfest
Differential Revision: D17611123
fbshipit-source-id: b621a422480b00eb2e339ff7542cc66c3ca5b8ec
Summary:
Change the `//scm/hg:hg` target to use an `sh_binary()` rule that invokes the
`:hg_rust` binary with the proper environment so it can find its dependencies,
rather than copying the binary and all of its dependencies into a new
subdirectory.
In dev mode builds the `hg_rust` binary isn't guaranteed to work anywhere
other than its original location, due to the way that dev mode builds use
`$ORIGIN` in the binary's `RPATH` setting. This happened to work up until now
as the hg_rust binary did not have any separate libraries, but I plan to add
one on the `chg` library.
Reviewed By: quark-zju
Differential Revision: D17109104
fbshipit-source-id: ae8bb1126969f012d1d2fb7d04e80867a310b9a8
Summary:
Given a list of changesets, a source and a target repo id, this query tells
you the first changeset from the list, which comes from the source repo and
has a synced equivalent in the target repo. Here, the word "first" refers to
the order of changesets in the provided list.
The naming is a little unfortunate, as the use case for this query is to
find the *last* (meaning newest) changeset that had been synced. But I wanted
the name to reflect this particular fn's behavior, rather than the intended
use.
Reviewed By: farnz
Differential Revision: D17600505
fbshipit-source-id: ea7336dcf839302e7b9ca6e1bc2ee88a3a1d99a9
Summary:
It gives us a nice ability to classify push errors and log them separately.
For example, we can distinguish pushrebase conflicts and hook failures
Reviewed By: farnz
Differential Revision: D17602576
fbshipit-source-id: 014197109056e044f59c397f7ccd17c93fadfe91
Summary: I'd like to get this in Scuba to do a bit more reporting there.
Reviewed By: farnz
Differential Revision: D17600786
fbshipit-source-id: 90d34362b5212abf3ff6642f55e57a997f86006f
Summary:
The method name was incorrectly added to the ODS key, rather than the repo
name. Fix this. :-)
Reviewed By: krallin
Differential Revision: D17600416
fbshipit-source-id: 203f3be428b15a0b7d1cce90dd85a3932311c89c
Summary:
Previously new bookmarks from overlay were ignored. This diff fixes it.
Also new sync job tests didn't have replay hook enabled. This diff fixes it and removes a bit of code duplication
Reviewed By: krallin
Differential Revision: D17597561
fbshipit-source-id: 686499e94f32b7b39fd00fa1af449bd1b456775e
Summary:
This is nice when running locally to know when we're done with setting up
cachelib and such.
Reviewed By: farnz
Differential Revision: D17569829
fbshipit-source-id: adfe5944991c8842a459df8606d9d81c2dcd02de
Summary:
After this diff new sync job (the one that generates bundles instead of reusing
existing) should be feature complete i.e. it should support the same pushes
as normal sync job.
One noteable refactoring is adding BookmarkChange enum. Previously we were passing and converting `from_` and `to_` parameters down the stack and also doing conversion in the middle, and it was easy to mix them up and use `from_hg_cs_id` instead of `to_hg_cs_id` and vice versa. My hope is that with this enum isolates most of the `from_` and `to_` shenanigans in one places so that's it's easier to keep track of it.
Reviewed By: krallin
Differential Revision: D17546808
fbshipit-source-id: 1b5e2553f80b33996ec361e89b01cb55ec811ec7
Summary:
After this diff sync job [1] starts to keep track of bookmark on hg server.
There are two reasons for that:
1) Some pushes just move a bookmark i.e. they don't push any new commits.
However without knowing hg server state it's impossible to tell apart normal push vs
bookmark-only push and generate correct bundles for it (note that the actual
support for bookmark-only pushes will be added in the next diffs, this diffs
just tracks hg sever bookmarks).
2) There are force pushes that move a bookmark to a completely new place AND
push new commits. Without knowing the state of hg server the sync job might try
to push too many commits.
To track the bookmarks they are fetched from hg server on sync job start up,
and later updated and the sync job progresses. To fetch bookmark from hg server
a separate listserverbookmarks extension is used.
If bundles are prepared together (see --bundle-prefetch option) then we also need to have bookmark overlay
i.e. bookmarks that were modified by the previous bundles in the same batch.
Note that combine-bundles is removed since it hasn't been used at all, include tw specs.
[1] It only keeps track if the job needs to regenerate bundles
Reviewed By: krallin
Differential Revision: D17525983
fbshipit-source-id: 528949ad4ad57ae51ad68fced9caf7256a057ba3
Summary: This is the bare minimum to ask the remapping table for any values it has. Combined with other admin tool commands, you can use this to track down a synced commit.
Reviewed By: StanislavGlebik
Differential Revision: D17554261
fbshipit-source-id: f5f967a55182b614b68b5a8c1421401921c9a268
