Commit Graph

5610 Commits

Author SHA1 Message Date
Jun Wu
295047e02a tests: remove some rarely used tests
Summary:
Those tests show up as "skipped". I removed some rarely used ones:
- test-check-jshint: we seldom write javascript.
- test-check-shbang: a slow test that does not do much.
- test-chg: chg is being tested in other tests by default.
- test-docker-packaging: probalby already broken, we don't use docker.
- test-verify-repo-operations: a slow test that is too expensive to run with
  limited value.

Reviewed By: markbt

Differential Revision: D21989103

fbshipit-source-id: 8080ad68dc75b59f87336b94af38c8231f4f10f4
2020-06-15 11:36:07 -07:00
Viet Hung Nguyen
d622ecb06a mononoke/bonsai_git_mapping: add CoreContext to BonsaiGitMapping functions
Summary: Log sql accesses in bonsai_git_mapping: added (core: &CoreContext) arguments to BonsaiGitMapping trait functions. Increment counters in functions where we have sql requests (e.g SQLReadsReplica). Updated tests covering BonsaiGitMapping with mock context object.

Reviewed By: StanislavGlebik

Differential Revision: D22043831

fbshipit-source-id: d05b07e262a8b7494d2ae46d5660d1c0695619ae
2020-06-15 11:23:23 -07:00
Mateusz Kwapich
0a915a26a8 fix the problem with updating skiplists after trimming
Summary:
After being trimmed down to store only a single edge the skiplist update
algoritm was working incorrectly. This diff introduces a way of fixing things
up.

Reviewed By: StanislavGlebik

Differential Revision: D21976995

fbshipit-source-id: 3bc4678011b332419d9f6f7b983a33fa066674bf
2020-06-15 09:58:15 -07:00
Arun Kulshreshtha
7d72408ade edenapi_server: return streaming responses
Summary: Make most of the handlers on the EdenAPI server return streaming responses.

Reviewed By: krallin

Differential Revision: D22014652

fbshipit-source-id: 177e540e1372e7dfcba73c594614f0225da3a10f
2020-06-15 09:11:09 -07:00
Mateusz Kwapich
33b5dd5c41 show problem with updating trimmed skiplists
Summary:
In production we trim skiplists to hold a single entry which causes
the update algorithm to behave incorrectly

Reviewed By: StanislavGlebik

Differential Revision: D21976996

fbshipit-source-id: 4946be11512ac9577bdd3320913fe092d513ecdd
2020-06-15 08:14:36 -07:00
Mateusz Kwapich
bc20c35b90 migrate ReachabilityIndex trait to async/await
Reviewed By: StanislavGlebik

Differential Revision: D21958617

fbshipit-source-id: 5acdfdc5e4dd9dc694a32bcf6441fdb4ca0f5776
2020-06-15 08:14:36 -07:00
Mateusz Kwapich
a6a77a1466 make LeastCommonAncestorsHint async
Reviewed By: StanislavGlebik

Differential Revision: D21958620

fbshipit-source-id: 597ed2069eb0b8f7668454548d4a347e51324024
2020-06-15 08:14:36 -07:00
Mateusz Kwapich
fe0ccc5b08 migrate fetch_skiplist_index to async/await syntax
Reviewed By: StanislavGlebik

Differential Revision: D21955570

fbshipit-source-id: 3d260520618af8c19d1cc3ffe20d73f3c676ed47
2020-06-15 08:14:36 -07:00
Mateusz Kwapich
0071d1803c migrate add_node to async/await syntax
Reviewed By: StanislavGlebik

Differential Revision: D21951752

fbshipit-source-id: 7de23875e51196018227729966701a4980a7c89a
2020-06-15 08:14:35 -07:00
Stanislau Hlebik
975ce02007 rust/thrift: add an option to stop processing requests if client disconnected, take 2
Summary:
Support for this was added in D20763778, however they were reverted later in
D21762641. Turned out that all cancelled requests were counting as still
"active" even though they were cancelled a long time ago. It accumulated over
time, and at some point (to be precise, at ~65K of "active" connections) thrift
server started to reject new connections.

The problem was that original diff D20763778 didn't delete ReplyState
structure, which had pointers to a few other C++ structures whose destructor
decremented the number of active connections. To be precise:
1) https://fburl.com/diffusion/qxsu4dqe - ReplyState is allocated here, and was
previously deallocated here - https://fburl.com/diffusion/9fj9ksgx
2) ReplyState has a pointer to ResponseChannelRequest
(https://fburl.com/diffusion/6au1duav), which is in our case is an instance of
Cpp2Request, whose destructor decrements the number of active requests
(https://fburl.com/diffusion/ylp2108m).

This diff actually does two things. It adds back the changes that were added in
D20763778, and also adds a fix that explicitly deletes ReplyState structure.
For convinience, you can see just the fix in D22017238.

Previously we were deleting ReplyState in send_reply function, however now
freeReplyState function should be explicitly called from the Rust code, which
should also fix a memory leak we had when `processor.call()` method returned an error
(https://fburl.com/diffusion/ymnwc3jx).

This is quite unfortunate that we have to do a manual actions to clean
up the memory - I'll try to improve it in the next diff.

Reviewed By: markbt

Differential Revision: D22018113

fbshipit-source-id: 2e9762bf7300129ad9e58e39fb93fca6f9505159
2020-06-15 07:17:12 -07:00
Mark Thomas
f80205bc96 repo_client: log infinitepush commits to scribe
Summary:
Add logging of infinitepush (draft) commits to a separate scribe category.

The logging will also include the username and hostname of the pusher.  Since
this code is shared with the public commits scribe logging, that logging will
also gain this information.

Reviewed By: farnz

Differential Revision: D21742656

fbshipit-source-id: bdbfd14db9e8aae190c634ac4bfff35b3f62bbe4
2020-06-15 06:32:55 -07:00
Alex Hornby
ea910afed0 mononoke: remove unused import
Summary: Found this wasn't used while looking at prefixblob usages

Reviewed By: farnz

Differential Revision: D22034647

fbshipit-source-id: a0d7dc02dbee9fdf685df464539323d4c200a818
2020-06-15 04:22:08 -07:00
Mateusz Kwapich
89341f08bb migrate process_frontier to async/await syntax
Summary: incremental step on the path to modernization of this module

Reviewed By: StanislavGlebik

Differential Revision: D21937310

fbshipit-source-id: 6ae5f9eb43355efee35705cce96736bfcb19c8ca
2020-06-15 02:55:35 -07:00
Mateusz Kwapich
fc05b8d667 migrate query_reachability internal implementation to async/await syntax
Summary:
NOTE: this diff doesn't change the public API of the trait - that's why we do
`compat()` and `boxify()`. This change will follow later in the stack.

Reviewed By: StanislavGlebik

Differential Revision: D21937308

fbshipit-source-id: 11bef4c5087a9ac06e209685d3281483b804c394
2020-06-15 02:55:35 -07:00
Mateusz Kwapich
592e696146 remove one of ancestry check implementations
Summary:
I'm not sure why but D14405696 added this `is_ancestor` method with a
duplicate ancestry check implementation which:
 * has different behaviour when `ancestor == descendant`
 * doesn't log to ODS

Let's just settle on using one implementation now.

Reviewed By: StanislavGlebik

Differential Revision: D21937312

fbshipit-source-id: 0f4e2390cb0b984f92cd4fe645d9313fa4bba70e
2020-06-15 02:55:35 -07:00
Mateusz Kwapich
a56afca442 migrate lazy_index_node to async/await syntax
Summary: incremental step on the path to modernization of this module

Reviewed By: StanislavGlebik

Differential Revision: D21937311

fbshipit-source-id: 9db84c7cb166084edfc5a0638c1c884964f9984c
2020-06-15 02:55:34 -07:00
Mateusz Kwapich
86d3e87b16 migrate find_nodes_to_index to async/await syntax
Summary: incremental step on the path to modernization of this module

Reviewed By: StanislavGlebik

Differential Revision: D21937309

fbshipit-source-id: 61b1b4302782cea35d3f65e6fe982f530293dd81
2020-06-15 02:55:34 -07:00
Durham Goode
5a83869eb1 thrift: add serde to generated thrift types
Summary:
This will let us serialize and deserialize thrift structures from
non-thrift sources. In my particular case I'll be using it to deserialize json
into hg client config thrift structures that are shared with the backend.

Reviewed By: quark-zju

Differential Revision: D21875371

fbshipit-source-id: 6a85f28e69f2eeb375f6639ed1dcd3c1cd957d99
2020-06-12 16:51:52 -07:00
Arun Kulshreshtha
a5d36bf798 edenapi_server: Remove 'pretty' param from repos endpoint
Summary: The intention of this parameter was to produce human-readable JSON for tests and for manual debugging. In practice, given that the JSON response here is just a (typically short) list, there isn't really much value in prettifying. We weren't using this anyway, so             let's remove it to simplify the code.

Reviewed By: xavierd

Differential Revision: D22020784

fbshipit-source-id: 09290cc667cfdb194693e6c05858af2f8905f88e
2020-06-12 16:41:30 -07:00
Arun Kulshreshtha
8f3c80fbe5 edenapi_server: Add explicit file and tree handler functions
Summary: Add wrapper functions around `data::data` for files and trees. The intention is to make the code more easily searchable, and also to establish a consistent naming pattern across all the handler functions.

Reviewed By: xavierd

Differential Revision: D22020783

fbshipit-source-id: 5e7c63ff8f366b0c6aa4412d5d04d17ca6cbc6b9
2020-06-12 16:41:30 -07:00
Arun Kulshreshtha
2110dd9c26 edenapi_server: use macro to reduce handler boilerplate
Summary: Reduce the amount of repeated code in the handlers module by defining a macro to wrap async handler functions.

Reviewed By: krallin

Differential Revision: D22015557

fbshipit-source-id: 276dad4112e7019f8d5f5fa74243747a0104e1ca
2020-06-12 13:01:20 -07:00
Xavier Deguillard
8b9401b0b2 pyerror: format the original error, not the downcasted one
Summary:
The anyhow error contains the context for the error which will lead to better
error message for the user (and us). What the code previously did was simply
using the Debug trait to print the error, and thus was missing context.

Reviewed By: DurhamG

Differential Revision: D21985745

fbshipit-source-id: 31c603d7f42e79a360541f39e4aaf0fcfbb9a14f
2020-06-12 12:38:06 -07:00
Ailin Zhang
ec906443ea print fsck to stdout
Summary: This diff passes a callback function `progressCallback` from `EdenServer` to make `OverlayChekcer` print fsck messages to stdout.

Reviewed By: fanzeyi

Differential Revision: D21966060

fbshipit-source-id: 317b2c8954c718b51b5295c9f8d7698eb203906e
2020-06-12 12:34:55 -07:00
Arun Kulshreshtha
48544b0233 edenapi_server: add ErrorKind
Summary: Add an `errors` module containing an `ErrorKind` implemented via `thiserror`. All errors in the server's HTTP handles been annotated with an appropriate `ErrorKind` via `anyhow::Context`. Notably, there is specialized handling of `MononokeError`s from the `mononoke_api` crate via a new `MononokeErrorExt` trait.

Reviewed By: krallin

Differential Revision: D22012237

fbshipit-source-id: 5f6948696e4741816636f1c6fee12069ee1a4b32
2020-06-12 12:16:52 -07:00
Arun Kulshreshtha
63ffc2d838 mononoke_api: add Send + Sync + 'static bounds to HgDataContext and HgDataId
Summary:
Add trait bounds to ensure that implementors of these trait can be
safely used across threads. This is particularly useful when using these traits
in generic async code -- async code typically requires these trait bounds on
data captured in combinator closures, etc.

Reviewed By: krallin

Differential Revision: D22008466

fbshipit-source-id: 48eac59ca62debc9bd293261f6079a4fa134f331
2020-06-12 12:16:51 -07:00
Arun Kulshreshtha
19b249d2b7 edenapi_server: refactor utils module
Summary: Move the `util` module out of `handlers` and split it up into multiple submodules.

Reviewed By: krallin

Differential Revision: D22008148

fbshipit-source-id: 11b286fdef8f9d93fcc10a283ec1b2f42e9c8c04
2020-06-12 12:16:51 -07:00
Arun Kulshreshtha
14846656f5 edenapi_types: restructure crate
Summary:
Note: Although this diff looks big, it's all just cutting and pasting code.

Restructure the crate so that the types relevant to each EdenAPI endpoint are grouped in their own module. This means that authors of new endpoints can just add a new module and put their types there, rather than having to make edits throughout the crate.

Reviewed By: quark-zju

Differential Revision: D21983470

fbshipit-source-id: 4901662dc6b5f0a1feca1ab8d914f394114ef233
2020-06-12 12:16:51 -07:00
Arun Kulshreshtha
b7aafa1df9 edenapi_types: add HistoryResponseChunk
Summary: Change `HistoryResponse` so that instead of repeatedly sending the file path alongside every history entry, we group the history entries by file and send the path once per group. This will prevent the server from transmitting redundant information.

Reviewed By: quark-zju

Differential Revision: D21982558

fbshipit-source-id: f1c5d2573c97940c7bf1645ed7fef6e1887c0d42
2020-06-12 12:16:51 -07:00
Chad Austin
6185a90dc6 fix warnings on macOS
Summary: Fix a couple warnings in the macOS build.

Reviewed By: genevievehelsel

Differential Revision: D21992085

fbshipit-source-id: 578adb3b95d29177740debbce4dc4626ba9eed64
2020-06-12 10:28:28 -07:00
Lukas Piatkowski
07e0427eb4 mononoke: make segmented_changelog OSS buildable
Reviewed By: krallin

Differential Revision: D21980974

fbshipit-source-id: 60e8f0e78b2bd654e5109b39c85f08ee5b3e9490
2020-06-12 06:54:32 -07:00
Thomas Orozco
51623fbdbd mononoke/tests: disable mutation.record everywhere
Summary:
This is consistent with what is being done for now in hg for tests that haven't been migrated
to modern configurations yet, and ensures we get stable commit hashes in our tests: D21899139. It's already explicitly turned on on tests that want it.

In the future, this should probably be updated to use "modern configs" like the
Mercurial tests do.

Reviewed By: ikostia

Differential Revision: D22016705

fbshipit-source-id: b27f6423bf4ec5244ef3ce2e7676306165a331a8
2020-06-12 04:17:18 -07:00
Thomas Orozco
304701f890 mononoke/tests: remove expecting a message that is gone
Summary: This message is gone we shouldn't expect it anymore: D21913608

Reviewed By: ikostia

Differential Revision: D22016684

fbshipit-source-id: 97d86e9750e775c1bb3a1e75939f506cd35851c0
2020-06-12 04:17:17 -07:00
Thomas Orozco
01a84fabe4 mononoke: turn off mutation in test-commitcloud
Summary:
This test broke when this got turned on for all tests (D21899139). It's not
enabled for other commit cloud tests there, so let's be consistent.

Reviewed By: ikostia

Differential Revision: D22016686

fbshipit-source-id: 5f4385b60fd31c89e335e971f262da1226f32254
2020-06-12 04:17:17 -07:00
Thomas Orozco
cb64897c70 infinitepush: only warn about mutation server support if entries are to be sent
Summary:
In the Mononoke integration tests (which are .t tests), we've started seeing
this waring show up.

We probably shouldn't have mutation enabled for most of those tests anyway
(save for the tests that actually exercise mutation), so I'm turning that off.
That said, we'd still see the warning. So, this diff refactors mutation a
little bit to only warn if we do have something to send to the server.

Reviewed By: ikostia

Differential Revision: D22016685

fbshipit-source-id: e9e8dbcc44a0fa048dca3ca7ce0961631058a2d6
2020-06-12 04:17:17 -07:00
Jeremy Fitzhardinge
d03da109c7 mononoke: fix some little lints
Reviewed By: StanislavGlebik

Differential Revision: D22004427

fbshipit-source-id: 1418f91a3322ef3a3ff620c4e17745fc765dc971
2020-06-12 01:33:20 -07:00
Arun Kulshreshtha
b462c0575c edenapi_server: remove unused futures-old dependency
Summary: Last usage was removed in D20342689.

Reviewed By: xavierd

Differential Revision: D22011195

fbshipit-source-id: 58455ad3dc77438704e80c69eb728a319ee71a39
2020-06-11 22:28:53 -07:00
Genevieve Helsel
ca1da512bf store force restart action as its own flag
Summary: In its current state, FORCE is a restart mode that cannot be combined with other modes (specifically `graceful`). Removing this coupling from `force` will allow us to at first attempt a graceful restart, but if the daemon is not healthy, then do a force restart, all within one call.

Reviewed By: wez

Differential Revision: D21873166

fbshipit-source-id: 079a777ff6e7b219198cbef15335244b9865f5f3
2020-06-11 17:40:51 -07:00
Arun Kulshreshtha
5be68125a7 gotham_ext: Add StreamBody
Summary:
Move the LFS server's `StreamBody` into `gotham_ext`, along with some changes to decouple it from the LFS server.

In particular, the `Content-Length` header and post-request `Sender` have been made optional fields that can be set via a builder-style interface. The LFS server's `StreamBody` has been renamed to `LfsStreamBody` and is now a thin wrapper around `gotham_ext`'s `StreamBody` that preserves the old behavior.

Reviewed By: krallin

Differential Revision: D21988855

fbshipit-source-id: a9bf9c04bb791388d761fc705ebc38472a713b65
2020-06-11 13:03:24 -07:00
Stefan Filip
f586f46ca5 mononoke: move replication_lag utilities from blobstore_healer to sql_ext
Summary: I would like to use these utilities when building segmented changelog.

Reviewed By: krallin

Differential Revision: D21876432

fbshipit-source-id: 9022627e224bfcb155b47d696371d24e538e6f39
2020-06-11 12:38:50 -07:00
Stefan Filip
c06de91150 mononoke: add tracking for SQL connection usage
Summary: Incrementing CoreContext::perf_counters().

Reviewed By: krallin

Differential Revision: D21875162

fbshipit-source-id: 0187034c3004efb86d74ce23ddf61a2b469efc61
2020-06-11 12:38:49 -07:00
Stefan Filip
754c1d00c0 mononoke: add counters to segmented changelog calls
Summary: Title.

Reviewed By: krallin

Differential Revision: D21875163

fbshipit-source-id: e7c50e30d10a4d192edb082c0cabe290c4e6d29a
2020-06-11 12:38:49 -07:00
Stefan Filip
0bb787d892 mononoke: add batch query capability to segmented changelog IdMap
Summary:
Useful for keeping the insert logic working nicely even when we double insert
or have other inconsistencies.

Reviewed By: krallin

Differential Revision: D21853048

fbshipit-source-id: 958cbc0435f330749c5aae1ea0f41ecd03b01468
2020-06-11 12:38:48 -07:00
Mark Thomas
76bff4c32d segmented_changelog: fix warnings
Reviewed By: sfilipco

Differential Revision: D21998893

fbshipit-source-id: f0b8d82ec4937263fbe0db2e626a3847118e5b08
2020-06-11 10:55:31 -07:00
Mark Thomas
d3a02aa301 scs_server: use service_identity for repo_create_commit
Summary:
If the caller provides a service identity in `repo_create_commit`, use that to obtain
write access to the repository.

Reviewed By: StanislavGlebik

Differential Revision: D21995724

fbshipit-source-id: ba218e392b979938d598a37494b0ffe496f28722
2020-06-11 10:44:52 -07:00
Mark Thomas
4b76a1564d mononoke_api: remove legacy functions
Summary: Now that apiserver is gone, we can remove the legacy parts of mononoke_api.

Reviewed By: krallin

Differential Revision: D21976615

fbshipit-source-id: 139d7ebd4a7185693f6a4fe4e5c234c143f18498
2020-06-11 10:44:52 -07:00
Mark Thomas
f261469a2f scs_server: implement repo_move_bookmark
Summary: Implement repo_move_bookmark by calling the appropriate method in mononoke_api.

Reviewed By: StanislavGlebik

Differential Revision: D21904980

fbshipit-source-id: 537abe2241688f9c18ad279c169e30d9cee51c66
2020-06-11 10:44:51 -07:00
Mark Thomas
b0509f5419 mononoke_api: add move_bookmark method
Summary:
Add the `move_bookmark` method to `mononoke_api`.

This attempts to re-use code from `repo_client`.  This code isn't really designed to be called from this API, so the fit is very poor.  In a future diff we will fix this up.

The `repo_client` code for force-moving a bookmark does not support running hooks for that bookmark.  For now we will prevent API users from moving bookmarks that have hooks configured.  This will also be addressed in a future diff.

Reviewed By: krallin

Differential Revision: D21904979

fbshipit-source-id: 42bf840489e5b04f463c69c752bcaa5174630c21
2020-06-11 10:44:51 -07:00
Jun Wu
7728e73e01 mutation: do not auto migrate if proxy-obsstore is enabled
Summary:
There is no need to migrate if the obsstore is a proxy to the mutation store.
This avoids a crash reading the attribute `_all`.

Reviewed By: DurhamG

Differential Revision: D21999245

fbshipit-source-id: bfa6439982834c6835412fc175132526525af548
2020-06-11 10:00:42 -07:00
Xavier Deguillard
e2bb3e23d7 tests: do not enable mutation for test-common-commands-fb.t
Summary: The test breaks, let's not enable mutation for it.

Reviewed By: DurhamG

Differential Revision: D21998151

fbshipit-source-id: 9c06485287d906837444f39faf90886933decc9b
2020-06-11 09:44:24 -07:00
Kostia Balytskyi
87970c4168 tests: fix broken test-infinitepush-bookmarks-disabled.t
Reviewed By: markbt

Differential Revision: D21997584

fbshipit-source-id: b2a1f0daf540911cdde9735a53f4d2b6d3d6984e
2020-06-11 09:22:28 -07:00