Commit Graph

5866 Commits

Author SHA1 Message Date
Jun Wu
adcf846f2f test-reduction: use modern configs
Summary:
Enable remotenames and narrow-heads.

Local bookmarks are replaced by remote bookmarks, causing the test change.

Reviewed By: krallin

Differential Revision: D22200500

fbshipit-source-id: aeee528d1766e0642c12e78a6c1a50cadc8a579a
2020-06-29 13:00:07 -07:00
Jun Wu
d1da1d70c1 test-push-redirector-sync-job: use modern configs
Summary:
Enable remotenames and narrow-heads.

The commits become 'draft' because there are no remote bookmarks.

Reviewed By: krallin

Differential Revision: D22200514

fbshipit-source-id: 04d0befa7c22756e936a28ffdcdf1305057cf062
2020-06-29 13:00:07 -07:00
Jun Wu
2875942761 test-infinitepush: use modern configs
Summary:
Enable remotenames and narrow-heads.

The test was migrated cleanly. The only change is that local bookmarks are
replaced by remote bookmarks.

Reviewed By: krallin

Differential Revision: D22200510

fbshipit-source-id: f5b8cd2ed125e9fc4e5daac897851d91fef5693f
2020-06-29 13:00:07 -07:00
Jun Wu
6ecf255fcf test-infinitepush-mutation: use modern configs
Summary:
Enable remotenames and narrow-heads.

Local bookmarks are replaced by remote bookmarks.

Reviewed By: krallin

Differential Revision: D22200503

fbshipit-source-id: 41ac4f4f606011dcaf6d0d9867b01fb77b9a79d8
2020-06-29 13:00:07 -07:00
Jun Wu
95cf4a2a39 test-infinitepush-hydrated: use modern configs
Summary:
Enable remotenames and narrow-heads.

Phase exchange is gone because of narrow-heads.
The remtoenames extension was written suboptimally so it issued a second
bookmarks request (which, hopefully can be removed by having selective
pull everywhere and migrate pull to use the new API).

Reviewed By: krallin

Differential Revision: D22200506

fbshipit-source-id: c522bb9fc1396d813e0f1f380c4290445bab3db3
2020-06-29 13:00:07 -07:00
Jun Wu
ffde9f50e9 test-infinitepush-commits-disabled: use modern configs
Summary:
Enable remotenames and narrow-heads. The `master_bookmark` is no longer a local
bookmark in the client repo.

Reviewed By: krallin

Differential Revision: D22200513

fbshipit-source-id: bc3c1715ce21f45a35bc67148eb00e44944bea6e
2020-06-29 13:00:06 -07:00
Jun Wu
8ecb79a921 test-gettreepack-sparse-update: use modern configs
Summary: Enable remotenames and narrow-heads.

Reviewed By: krallin

Differential Revision: D22201083

fbshipit-source-id: 585dff69db9dd725c8fa1090d47c85b150f979da
2020-06-29 13:00:06 -07:00
Jun Wu
439c029007 test-gettreepack-designated-nodes: use modern configs
Summary:
Enable remotenames and narrow-heads. The server gets one more request from
remotenames.

Reviewed By: krallin

Differential Revision: D22200502

fbshipit-source-id: 26bc28b19438c7be4a19eae6be728c83b113f822
2020-06-29 13:00:06 -07:00
Jun Wu
93318c255b test-bookmark-hg-kind: use modern configs
Summary:
Enable remotenames and narrow-heads. The client gets remote bookmarks instead
of local bookmarks during clone and phases are decided by remote bookmarks.

Reviewed By: krallin

Differential Revision: D22200515

fbshipit-source-id: 12a9e892855b3a8f62f01758565de5f224c4942b
2020-06-29 13:00:06 -07:00
Jun Wu
8bde7d1316 tests: show remotenames in tglogpnr
Summary:
Change the template to show remote bookmarks, which will be more relevant once
we migrate to modern configs. Namely, phases will be decided by remote bookmarks.

The named branches logic was mostly removed from the code base. Therefore
drop the `{branches}` template.

Reviewed By: StanislavGlebik

Differential Revision: D22200512

fbshipit-source-id: 8eca3a71ff88b8614023f4920a448156fcd712d5
2020-06-29 13:00:06 -07:00
Jun Wu
dbd29b7d06 tests: turn on narrow-heads for some tests
Summary: With narrow-heads, the phase exchange step is skipped.

Reviewed By: krallin

Differential Revision: D22200504

fbshipit-source-id: 6ab366e7e68eb3b82f52acaa8f488747435e0ecf
2020-06-29 13:00:06 -07:00
Jun Wu
889beacdf1 tests: enable narrow-heads for Mononoke tests
Summary:
Most tests pass without changes. Some incompatible tests are added to the
special list.

Reviewed By: krallin

Differential Revision: D22200505

fbshipit-source-id: 091464bbc7c9c532fed9ef91f2c955d6e4f2df0b
2020-06-29 13:00:06 -07:00
Stanislau Hlebik
04ce32014d mononoke: log pushed commits to scribe
Summary: This is the final diff of the stack - it starts logging pushed commits to scribe

Reviewed By: farnz

Differential Revision: D22212755

fbshipit-source-id: ec09728408468acaeb1c214d43f930faac30899b
2020-06-29 12:15:22 -07:00
Stanislau Hlebik
6fd54d6b22 mononoke: don't fail if logging to scribe failed
Summary:
Failing push if we failed to log to scribe doesn't make a lot of sense. By that
time the ship has sailed - commit has already been pushed and by failing the
request we can't undo that. It will just create an annoyance by whoever is
pushing.

Instead let's log it to scuba

Reviewed By: farnz

Differential Revision: D22256687

fbshipit-source-id: 2428bbf1db4cef6fa80777ad65184fab1804fa9c
2020-06-29 12:15:22 -07:00
Stanislau Hlebik
8a137ae922 mononoke: add Scribe
Summary:
At the moment we can't test logging to scribe easily - we don't have a way to
mock it. Scribe are supposed to help with that.

They will let us to configure all scribe logs to go to a directory on a
filesystem similar to the way we configure scuba. The Scribe itself will
be stored in CoreContext

Reviewed By: farnz

Differential Revision: D22237730

fbshipit-source-id: 144340bcfb1babc3577026191428df48e30a0bb6
2020-06-29 12:15:22 -07:00
Jun Wu
4902a3300c tests: enable narrow-heads by default
Summary: Many tests are incompatible. But many are passing.

Reviewed By: kulshrax

Differential Revision: D22052475

fbshipit-source-id: 1f30ac2b0fe034175d5ae818ec2be098dbd5283d
2020-06-29 11:29:04 -07:00
Jun Wu
d8835e90a2 test-globalrevs: disable zstore commit data
Summary: This improves test compatibility.

Reviewed By: kulshrax

Differential Revision: D22174995

fbshipit-source-id: 27b4690567699004163b199b5ccbaf4f8850cd80
2020-06-29 11:29:04 -07:00
Jun Wu
0d20e20244 exchange: skip phase exchange if narrow-heads is on
Summary:
With narrow-heads, phases are not tracked by remotenames. There is no need to
do a separate phase exchange.

Reviewed By: kulshrax

Differential Revision: D22130170

fbshipit-source-id: 6ee8cda338253aea5df09e779ecaf3d29910f622
2020-06-29 11:29:04 -07:00
Jun Wu
6554961897 tests: replace 'phase --public' with 'debugmakepublic'
Summary: This makes narrow-heads migration smoother.

Reviewed By: ikostia

Differential Revision: D22052478

fbshipit-source-id: 83047ad6da4de2034c917bfd5be47a2f32d5cc3d
2020-06-29 11:29:04 -07:00
Jun Wu
b12980f84c commands: add debugmakepublic
Summary:
Many tests want to change commit phase to public. We don't want to leak this
feature to end-users. Therefore make a debug command.

Reviewed By: kulshrax

Differential Revision: D22052481

fbshipit-source-id: 4ce6a61120da0d74374465e97ac0ec0ff9638088
2020-06-29 11:29:04 -07:00
Jun Wu
4bfc5ea5d4 test-fb-hgext-undo: remove the test
Summary:
The test depends on too much details of the obsstore and is hard to
migrate to mutation. The undo command in production actually uses
a different code path (with narrow-heads, it rewrites metalog directly),
which is somewhat covered by test-undo-narrow-heads. Ideally we extend
the undo test with more cases under modern configs. For now let's just
remove the test with legacy configs to unblock migration progress.

Reviewed By: kulshrax

Differential Revision: D22174961

fbshipit-source-id: 57baf5e6f59938069b44a5db8c5ae1662f094f84
2020-06-29 11:29:04 -07:00
Jun Wu
50ed524e78 test-rebase-obsolete: remove the test
Summary:
It has too many dependencies on legacy components that it's too hard to
migrate.  `test-rebase-mutation` is the test that is more relevant.

Reviewed By: kulshrax

Differential Revision: D22174996

fbshipit-source-id: 40889bc2f80021839ce0ffe13098404e8aa97fbb
2020-06-29 11:29:04 -07:00
Jun Wu
2da2bfd958 test-commitcloud-sync-oscillation: remove the test
Summary:
The test is about syncing with obsmarkers. We no longer use obsmarkers so the
test is no longer relevant.

Reviewed By: kulshrax

Differential Revision: D22174999

fbshipit-source-id: 7615852b734319bc838b238f4c9b54252ba5c355
2020-06-29 11:29:04 -07:00
Jun Wu
9bb4871a10 test-phases: remove the test
Summary:
The test is mostly about manipulating phases and phase exchange in bundle2.
Both of them are gone with narrow-heads, which makes phases immutable.
Practically, phases affect whether commits are immutable and that are covered
by individual tests (ex. amend tests check amend fails on public commits).

Reviewed By: kulshrax

Differential Revision: D22174955

fbshipit-source-id: cabc0de552f0ede54a23a6e6eec47b14c2a863b9
2020-06-29 11:29:03 -07:00
Jun Wu
46da14cc76 hiddenerror: remove extension
Summary:
With narrow-heads (enabled in production and most tests) hiddenerror will
never happen. Remove the extension.

Reviewed By: kulshrax

Differential Revision: D22174987

fbshipit-source-id: e3e3823d3da89dfd51b60f65f8e138ea723cc00f
2020-06-29 11:29:03 -07:00
Jun Wu
11e6a05277 test-bundle2-pushback: remove the test
Summary: The pushback behavior is already tested extensively in pushrebase tests.

Reviewed By: kulshrax

Differential Revision: D22130172

fbshipit-source-id: 7b3a5ab2792969b1dce57ae09406aef063450568
2020-06-29 11:29:03 -07:00
Jun Wu
741016fb77 test-flagprocessor: remove the test
Summary:
We don't use the flag processor framework for things other than LFS in the
non-Rust code. It is coupled with revlog, which is going away. Our LFS usage is
covered by other tests.

Reviewed By: kulshrax

Differential Revision: D22130168

fbshipit-source-id: 78a04df49961e2faca0def71daeec6a40e45c4b5
2020-06-29 11:29:03 -07:00
Jun Wu
c3124548eb tests: remove tests about legacy phase exchange
Summary:
We have narrow-heads on in production. Phases are decided by remote bookmarks
instead of exchanged separately. Remove the test since it's no longer relevant.

Reviewed By: sfilipco

Differential Revision: D22052477

fbshipit-source-id: c60382448d5e4d3d2d1225762b6c133873016881
2020-06-29 11:29:03 -07:00
Jun Wu
5bb0223a54 localrepo: fix headrevs ordering
Summary:
The return order of `index2.headsancestors` and `index.headrevs` are different.
Fix it. This affects the default commit to update to after `hg clone` in some
tests.

Reviewed By: sfilipco

Differential Revision: D22200496

fbshipit-source-id: 273201461c814241ce8dafa6f7ca00f72ea83eab
2020-06-29 11:29:03 -07:00
Jun Wu
e97f281c3e repo: hide narrow-heads migration message on repo creation
Summary:
If the repo is newly created, do not show the migration message.
This makes migration smoother.

Reviewed By: sfilipco

Differential Revision: D22052480

fbshipit-source-id: 4857bcb3e38094501eaa63a52f09d49853cf761b
2020-06-29 11:04:23 -07:00
Simon Farnsworth
7e9b8dd9e9 Remove last vestiges of Lua hooks from tests
Summary:
For Lua hooks, we needed to know whether to run the hook per file, or per changeset. Rust hooks know this implicitly, as they're built-in to the server.

Stop having the tests set an unnecessary config

Reviewed By: krallin

Differential Revision: D22282799

fbshipit-source-id: c9f6f6325823d06d03341f04ecf7152999fcdbe7
2020-06-29 10:03:22 -07:00
Lukas Piatkowski
b7b5c9e7cc shed/fbthrift_ext: move some useful fbthrift extensions to the shed
Reviewed By: farnz

Differential Revision: D22257111

fbshipit-source-id: b8f59cb0aa0d0e1100b5eb9c2a1f295ad6442650
2020-06-29 09:16:00 -07:00
Harvey Hunt
026710c2cd mononoke: Remove --config_path from server arguments
Summary:
D21642461 (46d2b44c0e) converted Mononoke server to use the
`--mononoke-config-path` common argument style to select a config path.

Now that this change has been running for a while, remove the extra logic in
the server that allowed it to accept both the deprecated `--config_path / -P`
and the new arg.

Reviewed By: ikostia

Differential Revision: D22257386

fbshipit-source-id: 7da4ed4e0039d3659f8872693fa4940c58bae844
2020-06-29 07:28:36 -07:00
Stanislau Hlebik
f55fa975a5 mononoke: fix unused imports
Reviewed By: krallin

Differential Revision: D22281331

fbshipit-source-id: 656ba6500193bd179d1e6cd1443de3e85d37c597
2020-06-29 03:22:11 -07:00
Kostia Balytskyi
fb3eea2b56 commit_validator: get rid of unneeded bookmark rewriting
Summary:
`get_entry_with_small_repo_mapings` is a function that turns a `CommitEntry`
struct into `CommitEntryWithSmallReposMapped` struct - the idea being that this
function looks up hashes of commits into which the original commit from the
large repo got rewritten (in practice rewriting may have happened in the small
-> large direction, but it is not important for the purpose of this job). So it
establishes a mapping. Before this
diff, it actually established `Large<ChangesetId> ->
Option<(Small<ChangesetId>, Option<BookmarkName>)>` mapping, meaning that it
recorded into which bookmark large bookmark was rewritten. This was a useless
information (as evidenced by the fact that it was ignored by the
`prepare_entry` function, which turns `CommitEntryWithSmallReposMapped` into
`EntryPreparedForValidation`. It is useless because bookmarks are mutable and
it is impossible to do historic validation of the correctness of bookmark
renaming: bookmarks may have been correctly renamed when commits where pushes,
but they may be incorrectly renamed now and vice-versa. To deal with bookmarks,
we have a separate job, `bookmarks_validator`.

So this diff stops recording this useless information. As a bonus, this will
make migration onto `LiveCommitSyncConfig` easier.

Reviewed By: StanislavGlebik

Differential Revision: D22235389

fbshipit-source-id: c02b3f104a8cbd1aaf76100aa0930efeac475d42
2020-06-29 01:48:52 -07:00
Kostia Balytskyi
b7dba9ff2f cross_repo_sync: expose get_commit_sync_outcome as a public fn
Summary: We need to be able to query `synced_commit_mapping` to understand which `version_name` was used to sync commits. That `version_name` will be needed to produce `CommitSyncConfig` by utilizing upcoming `LiveCommitSyncConfig` APIs. And `CommitSyncConfig` is needed to create `CommitSyncer`. So let's extract this fn out of `CommitSyncer`, as it's an independent functionality really

Reviewed By: farnz

Differential Revision: D22244952

fbshipit-source-id: 53e55139efd423174176720c8bf7e3ecc0dcb0d7
2020-06-27 04:42:54 -07:00
Jun Wu
01d2df5d5f repo: move remotenames heads logic to core
Summary:
This allows narrow-heads head calculation without the remotenames extension.

The test about related internal APIs is deleted. Other tests will cover the heads calculation indirectly.

Reviewed By: sfilipco

Differential Revision: D22052476

fbshipit-source-id: 99ad01b075b93e135d7f6c1bc27d837b19337726
2020-06-26 20:42:20 -07:00
Jun Wu
c6678914a2 repo: inline changelog._headrevs implementation
Summary: Move code around. This allows moving remotenames heads to core.

Reviewed By: sfilipco

Differential Revision: D22052479

fbshipit-source-id: 9c056e3b20c9b2cd9a0ff314181bb094ac834c82
2020-06-26 20:42:20 -07:00
Xavier Deguillard
a6a8c11b5a changelog: properly handle binary extras
Summary:
In configerator, extras might be non-utf8 and plain bytes. This causes 2
issues, the first one is when extracting the file list out of the change, the
other one is when dealing with the extras directly. For the first one, only
decode the file list, and for the second one, use the surrogateescape error
handling when decoding/encoding them.

Reviewed By: quark-zju

Differential Revision: D22260068

fbshipit-source-id: 19ef0248c69ea2b75bb91e59c3cdaed869110950
2020-06-26 19:20:29 -07:00
Yunus Rahbar
451d97d6ca Fix default commits value in diffidtonode
Summary:
The bad type on the default value is causing crashes when there are no "local:commits" fields on a phabricator version.

Created from Diffusion's 'Open in Editor' feature.

Reviewed By: chadaustin, grakkpl

Differential Revision: D22259485

fbshipit-source-id: b11a82b2b099aea373734af96cbd4834f5c5dcbc
2020-06-26 18:31:26 -07:00
Kostia Balytskyi
c01294e8d6 backsyncer_cmd: use LiveCommitSyncConfig
Summary:
This diff migrates `backsyncer_cmd` (the thing that runs in the separate backsyncer job, as opposed to bakcsyncer, triggered from push-redirector) onto `LiveCommitSyncConfig`. Specifically, this means that on every iteration of the loop, which calls `backsync_latest` we reload `CommitSyncConfig` from configerator, build a new `CommitSyncer` from it, and then pass that `CommitSyncer` to `backsync_latest`.

One choice made here is to *not* create `CommitSyncer` on every iteration of the inner loop of `backsync_latest` and handle live configs outside. The reason for this is twofold:
- `backsync_latest` is called form `PushRedirector` methods, and `PushRedirector` is recreated on each `unbundle` using `LiveCommitSyncConfig`. That call provides an instance of `CommitSyncer` used to push-redirect a commit we want to backsync. It seems strictly incorrect to try and maybe use a different instance.
- because of some other consistency concerns (different jobs getting `CommitSyncConfig` updates at different times), any sync config change needs to go through the following loop:
  - lock the repo
  - land the change
  - wait some time, until all the possible queues (x-repo sync and backsync) are drained
  - unlock the repo
- this means that it's ok to have the config refreshed outside of `backsync_latest`

Reviewed By: farnz

Differential Revision: D22206992

fbshipit-source-id: 83206c3ebdcb2effad7b689597a4522f9fd8148a
2020-06-26 13:40:31 -07:00
Arun Kulshreshtha
19c55df1fd http_client: add CLI for testing
Summary: Add a simple CLI to allow the HTTP client to be tested manually.

Reviewed By: quark-zju

Differential Revision: D22228930

fbshipit-source-id: 12fea3131ec6d8c3df4457fb74a09ea52f42c066
2020-06-26 13:24:21 -07:00
Arun Kulshreshtha
e054376937 http_client: allow setting CA cert path
Summary: Allow setting the CA certificate bundle, similar to the curl commands `--cacert` flag. This is required for integration tests since those servers use dummy certificates signed by a fake CA.

Reviewed By: quark-zju

Differential Revision: D22203833

fbshipit-source-id: 261e6c2904504c3a98f95b4a5b5b6ed24cb7402d
2020-06-26 13:24:21 -07:00
Arun Kulshreshtha
6e73978580 http_client: add new libcurl-based HTTP client
Summary:
Add a simple HTTP client library, based on libcurl. This crate is essentially an attempt to factor out the HTTP code from the Eden API client, since over time it had accumulated all of the pieces needed for a general-purpose HTTP client library. Factoring it out will help clean up the EdenAPI code base and allow code reuse by other crates that also need to work with HTTP.

This initial diff introduces the `Request` and `Response` types which can be used to build, send, and see the results of individual HTTP requests. In this diff, requests can only be made serially, but later in the stack it will be possible to run many requests concurrently (potentially multiplexed over the same connection in the case of HTTP/2).

The HTTP functionality in the EdenAPI client had very little unit test coverage (it relied primarily on Mononoke's integration tests). With this crate, I've added many unit tests (often involving mocked HTTP servers) to help ensure correctness.

Reviewed By: quark-zju

Differential Revision: D22157712

fbshipit-source-id: 3b0823ece26b19979980841727f1eefcf0519ad5
2020-06-26 13:24:21 -07:00
Xavier Deguillard
99ca6c74d9 win: do not ignore notification from .hg and .eden
Summary:
While suppressing notification should in theory be faster than getting
notification, it however causes a couple of issues due to the fact that EdenFS
is simply not creating inodes for any of the files in these directories.
Renaming a file in and out of these directories would for instance fail, due to
the inode not being present.

Since these directories are very low traffic overall and on Linux/macOS these
are not treated specifically (well, except for .eden, but that's a different
story), let's do the same on Windows.

Reviewed By: chadaustin

Differential Revision: D22250850

fbshipit-source-id: c13ed29faedc33c98b1a30227e44afc3f2c84c89
2020-06-26 10:41:28 -07:00
Nick Brekhus
f2435d8d4b metalog: implement compaction api
Summary: Compacts metalog by copying current root into new metalog and creating (or updating) a metalog-internal pointer file

Reviewed By: quark-zju

Differential Revision: D22100213

fbshipit-source-id: 7cea17dde46ac4fa2c84da873df68c536dca4119
2020-06-26 09:58:58 -07:00
Kostia Balytskyi
9a00efc973 cmdlib: expose test-instance and local-configerator-path
Summary:
Before this diff only the main Mononoke server binary was able to use fs-based
`ConfigStore`, which is pretty useful in integration tests.

Reviewed By: farnz

Differential Revision: D22256618

fbshipit-source-id: 493a064a279250d01469c9ff7f747585581caf51
2020-06-26 06:51:07 -07:00
Simon Farnsworth
7938a1957a Support BlobstoreWithLink in Sqlblob
Summary: We designed the schema to make this simple to implement - it's literally a metadata read and a metadata write.

Reviewed By: ikostia

Differential Revision: D22233922

fbshipit-source-id: b392b4a3a23859c6106934f73ef60084cc4de62c
2020-06-26 03:54:42 -07:00
Simon Farnsworth
b1c85aaf4b Switch Blobstore to new-style futures
Summary:
Eventually, we want everything to be `async`/`await`; as a stepping stone in that direction, switch the remaining lobstore traits to new-style futures.

This just pushes the `.compat()` out to old-style futures, but it makes the move to non-'static lifetimes easier, as all the compile errors will relate to lifetime issues.

Reviewed By: krallin

Differential Revision: D22183228

fbshipit-source-id: 3fe3977f4469626f55cbf5636d17fff905039827
2020-06-26 03:54:42 -07:00
Kostia Balytskyi
ef87f564bc add newtype for CommitSyncConfigVersion
Summary:
This is to avoid passing `String` around. Will be useful in one of the next
diffs, where I add querying `LiveCommitSyncConfig` by versions.

Reviewed By: krallin

Differential Revision: D22243254

fbshipit-source-id: c3fa92b62ae32e06d7557ec486d211900ff3964f
2020-06-26 02:45:26 -07:00