Summary:
This adds the ability to provide an infinitepush namespace configuration without actually allowing infinite pushes server side. This is useful while Mercurial is the write master for Infinite Push commits, for two reasons:
- It lets us enable the infinitepush namespace, which will allow the sync to proceed between Mercurial and Mononoke, and also prevents users from making regular pushes into the infinitepush namespace.
- It lets us prevent users from sending commit cloud backups to Mononoke (we had an instance of this reported in the Source Control @ FB group).
Note that since we are routing backfills through the shadow tier, I've left infinitepush enabled there.
Reviewed By: StanislavGlebik
Differential Revision: D16071684
fbshipit-source-id: 21e26f892214e40d94358074a9166a8541b43e88
Summary:
Added an option to control for which repositories should censoring be
enabled or disabled. The option is added in `server.toml` as `censoring` and it
is set to true or false. If `censoring` is not specified, then the default
option is set to true ( censoring is enabled).
By disabling `censoring` the verification if the key is blacklisted or not is
omitted, therefor all the files are fetchable.
Reviewed By: ikostia
Differential Revision: D16029509
fbshipit-source-id: e9822c917fbcec3b3683d0e3619d0ef340a44926
Summary:
- more tracing for potentialy large pieces of work
- removed some unnecessary tracing
Reviewed By: StanislavGlebik
Differential Revision: D15851576
fbshipit-source-id: 6686c00da56176cad43f72d1671e08eb8141110f
Summary: This is the first step towards per-repo control of whether pushes are allowed.
Reviewed By: StanislavGlebik
Differential Revision: D15519959
fbshipit-source-id: a0bb96bd995af7df0cef225c73d559f309cfe592
Summary:
This adds a sanity check that limits the count of matches in `list_all_bookmarks_with_prefix`.
If we find more matches than the limit, then an error will be returned (right now, we don't have support for e.g. offsets in this functionality, so the only alternative approach is for the caller to retry with a more specific pattern).
The underlying goal is to ensure that we don't trivially expose Mononoke to accidental denial of service when a list lists `*` and we end up querying literally all bookmarks.
I picked a fairly conservative limit here (500,000), which is > 5 times the number of bookmarks we currently have (we can load what we have right now successfully... but it's pretty slow);
Note that listing pull default bookmarks is not affected by this limit: this limit is only used when our query includes scratch bookmarks.
Reviewed By: StanislavGlebik
Differential Revision: D15413620
fbshipit-source-id: 1030204010d78a53372049ff282470cdc8187820
Summary:
This updates our receive path for B2xInfinitepush to create new scratch bookmarks.
Those scratch bookmarks will:
- Be non-publishing.
- Be non-pull-default.
- Not be replicated to Mercurial (there is no entry in the update log).
I added a sanity check on infinite pushes to validate that bookmarks fall within a given namespace (which is represented as a Regexp in configuration). We'll want to determine whether this is a good mechanism and what the regexp for this should be prior to landing (I'm also considering adding a soft-block mode that would just ignore the push instead of blocking it).
This ensures that someone cannot accidentally perform an infinitepush onto master by tweaking their client-side configuration.
---
Note that, as of this diff, we do not support the B2xInfinitepushBookmarks part (i.e. backup bookmarks). We might do that separately later, but if we do, it won't be through scratch Bookmarks (we have too many backup bookmarks for this to work)
Reviewed By: StanislavGlebik
Differential Revision: D15364677
fbshipit-source-id: 23e67d4c3138716c791bb8050459698f8b721277
Summary:
The idea was that the short description is used for mechanical summaries of the hook failures, and the long description is used for human-readable "how to handle this" forms.
Instead, we had a mixture of styles, plus only ever returning the short description. Change this to only ever return the long description and fix hooks so that the long description is meaningful
Reviewed By: StanislavGlebik
Differential Revision: D15537580
fbshipit-source-id: 6289c1c9786862db8190b4464a3133c0620eb09c
Summary:
Previously file hook would look up a file name from the root of the repo on
every hook run. So if we add or modify a lot of files in the same directory,
then the manifest for this directory will be parsed over and over again.
There is not need to do it, since we can just use file node id of the file.
This diff does exactly that - now HookFile will use FileNodeId instead of
(HgChangesetId + MPath) to look up a file content.
Reviewed By: krallin
Differential Revision: D15349196
fbshipit-source-id: 503109fc87ccf2659d481eeca165044baa440463
Summary:
As part of adding support for infinitepush in Mononoke, we'll include additional server-side metadata on Bookmarks (specifically, whether they are publishing and pull-default).
However, we do use the name `Bookmark` right now to just reference a Bookmark name. This patch updates all reference to `Bookmark` to `BookmarkName` in order to free up `Bookmark`.
Reviewed By: StanislavGlebik
Differential Revision: D15364674
fbshipit-source-id: 126142e24e4361c19d1a6e20daa28bc793fb8686
Summary: I'm going to be doing some work on this file, but it's not up-to-date with rustfmt. To minimize merge conflicts and simplify diff reviews, I ran that earlier.
Reviewed By: StanislavGlebik
Differential Revision: D15364675
fbshipit-source-id: 66e3d2287ffcaf93bc6ad17f0217517f0ddb9b2f
Summary:
There was a request about importing a GitHub repo into fbsource. While pushing
it to Mononoke with pushrebase disabled, the sync job broke because it can only
handle pushrebase pushes.
Before this diff, pushrebase has a repo-level config about whether dates need
to be rewritten. We definitely want "master" to have date rewritten turned on,
but not the imported commits. This diff adds logic to turn off date rewriting
for bookmarks by using the `rewrite_dates` config, to address the repo import
requirement.
Reviewed By: StanislavGlebik
Differential Revision: D15291030
fbshipit-source-id: 8dcf8359d7de9ac33f0af6f9ab3bcbac424323e4
Summary:
This migrates the internal structures representing the repo and storage config,
while retaining the existing config file format.
The `RepoType` type has been replaced by `BlobConfig`, an enum containing all
the config information for all the supported blobstores. In addition there's
the `StorageConfig` type which includes `BlobConfig`, and also
`MetadataDBConfig` for the local or remote SQL database for metadata.
Reviewed By: StanislavGlebik
Differential Revision: D15065421
fbshipit-source-id: 47636074fceb6a7e35524f667376a5bb05bd8612
Summary:
At the moment verify integrity script blocks master commits that do not pass
the check. Non-master commits are not blocked even if they don't pass the check
(for example, if they don't have a valid reviewer), but verify_integrity hook
still logs them to scuba.
Mononoke's verify_integrity was enabled only on master, meaning that the hook
won't run on non-master commits at all, and we won't log non-master commits to scuba.
This diff fixes it.
Reviewed By: farnz
Differential Revision: D15146725
fbshipit-source-id: ab432fb2eccae0fbcc10755f5c8447964c490285
Summary: Now it is possible to configure and enable/disable bookmark cache from configs
Reviewed By: StanislavGlebik
Differential Revision: D14952840
fbshipit-source-id: 3080f7ca4639da00d413a949547705ad480772f7
Summary:
We need this hook to replicate functionality of [disable-nonff.py](diffusion/OPSFILES/browse/master/chef/cookbooks/other/fb_mercurial_server/files/default/scripts/hooks/bin/disable-nonff.py;84ce14ceebcb6ef47b0c32feb0db742439e0ffb5$46-49)
I is now possible to restrict users from moving specified bookmark
Reviewed By: StanislavGlebik
Differential Revision: D14871808
fbshipit-source-id: e2f7e8b97f789cfaff2d76abc405e8bd1c6abdd8
Summary:
Previously, the hook_tailer would only print hook rejections if
the --debug flag was passed. Modify the tailer so that it always prints
out rejections.
Reviewed By: farnz
Differential Revision: D14799244
fbshipit-source-id: 3b2c5b00b6cfa54f6fa93c58406b1720876fd9d4
Summary: Update Rust toolchain to 1.33.0 with fixes to make our code compatible with 1.33.0.
Reviewed By: Imxset21, kulshrax
Differential Revision: D14608312
fbshipit-source-id: 2d9cf7d01692abaed32f9adffa0e5eb51cfacb4f
Summary:
This is a hook in mercurial, in Mononoke it will be part of the implementation. By default all non fastforward pushes are blocked, except when using the NON_FAST_FORWARD pushvar (--non-forward-move is also needed to circumvent client side restrictions). Additionally certain bookmarks (e.g. master) shouldn't be able to be moved in a non fastforward manner at all. This can be done by setting block_non_fast_forward field in config.
Pushrebase can only move the bookmark that is actually being pushrebased so we do not need to check whether it is a fastforward move (it always is)
Reviewed By: StanislavGlebik
Differential Revision: D14405696
fbshipit-source-id: 782b49c26a753918418e02c06dcfab76e3394dc1
Summary: There is no need to have an Option<Vec<XYZ>> as None can simply be represented by an empty vector. This makes these fields easier to use.
Reviewed By: StanislavGlebik
Differential Revision: D14405687
fbshipit-source-id: e4c5ba12a1e3c6a18130026af6814d54952da4d2
Summary: See D14279065, this diff is simply to clean up the deprecated code
Reviewed By: StanislavGlebik
Differential Revision: D14279210
fbshipit-source-id: 10801fb04ad533a80bb7a2f9dcdf3ee5906aa68d
Summary: This endpoint is also used in Mercurial now.
Reviewed By: StanislavGlebik
Differential Revision: D14303557
fbshipit-source-id: fe38b62d010de2846dcf800f93ba050d9c396873
Summary: The phabricator message parsing capability will be also used from check_unittests Rust hook, so it had to be extracted and adjusted to Rust standards.
Reviewed By: StanislavGlebik
Differential Revision: D14301561
fbshipit-source-id: 47b59527dfadd7b761f750825da52ffce14fdf21
Summary:
In this first diff check_unittests doesn't do much except it calls the interngraph endpoint with the proper app id and auth token.
Mocking and proper logic of this hook will be added in following diffs.
Reviewed By: StanislavGlebik
Differential Revision: D14089265
fbshipit-source-id: 17c4330691ba3ffde33668e59f8b1cad4a9242a1
Summary: New commits should be logged to scribe, these will be used to trigger the update for the hg clone streamfile.
Reviewed By: lukaspiatkowski
Differential Revision: D14022599
fbshipit-source-id: a8a68f12a8dc1e65663d1ccf1a5eafa54ca2daf0
Summary: This is to tell Mononoke whether to preserve bundle2 raw content or not.
Reviewed By: StanislavGlebik
Differential Revision: D14038588
fbshipit-source-id: f1781ed8b4aca7e37925e7f40aeb506af60d071a
Summary: This should be a big improvement for hooks that only care about file type or size and not the content.
Reviewed By: StanislavGlebik
Differential Revision: D14023307
fbshipit-source-id: 28856512b3092af9517079a9f379d23a9c4f3864
Summary: HgFileNodeId is a stronger typed id, so it is prefered to use it instead of HgNodeHash whenever it is identifying a filenode
Reviewed By: aslpavel
Differential Revision: D13986172
fbshipit-source-id: c0334652345acb868e86c38b8c0045e9c023c176
Summary: The Copy trait means that something is so cheap to copy that you don't even need to explicitly do `.clone()` on it. As it doesn't make much sense to pass &i64 it also doesn't make much sense to pass &<Something that is Copy>, so I have removed all the occurences of passing one of ouf hashes that are Copy.
Reviewed By: fanzeyi
Differential Revision: D13974622
fbshipit-source-id: 89efc1c1e29269cc2e77dcb124964265c344f519
Summary:
Repo name is used only be verify_integrity hook and even there the name that Mononoke provides is incorrect. Instead of Mononoke's `repo-RepositoryId(1234)` name the hook is interested in Mercurial's `fbsource` name.
HookConfig is a perfect way to pass such an arbitrary parameter so use it.
Reviewed By: StanislavGlebik
Differential Revision: D13964486
fbshipit-source-id: 94090e409d5206828364202ae62a37abc16e4a27
Summary:
This diff includes a bit of cleanup in hooks code, like:
- removing not used `repo_name` from `ctx.info`
- moving lua code from within `.rs` files to it's own `.lua` files
- addressing linter comments on lua code and
- reducing some copy-pasta between Changeset and File hook types
Reviewed By: StanislavGlebik
Differential Revision: D13895963
fbshipit-source-id: c867e060ad40e81fe0c81f30d14df59e75e129e6
Summary:
The main reason for doing it is to remove dependency on `BlobRepo` from hooks. Most of the `hooks` crate code needs from `BlobRepo` just a HgBlobChangeset type, which was moved to a separate crate in one of the previous diffs. There is just a small piece of code that depends on blobrepo, and it was moved in the separate crate.
Because of that changing anything in BlobRepo won't trigger rebuilding of most of the hooks crate.
Reviewed By: lukaspiatkowski
Differential Revision: D13878208
fbshipit-source-id: d74336e959282c176258c653d4c408854e1f1849
Summary:
Currently if a crate depends even on a single type from metaconfig then in
order to compile this trait buck first compiles metaconfig crate with all the
logic of parsing the configs.
This diff split metaconfig into two crates. The first one just holds the types for
"external consumption" by other crates. The second holds the parsing logic.
That makes builds faster
Reviewed By: jsgf, lukaspiatkowski
Differential Revision: D13877592
fbshipit-source-id: f353fb2d1737845bf1fa0de515ff8ef131020063
Summary: In this diff the configs are parsed from toml and passed around to hook's execution context. The actual usage of configs will be introduced in separate diff.
Reviewed By: StanislavGlebik
Differential Revision: D13862837
fbshipit-source-id: 60ac10aa9c25d224e703e1e55bef13dc481ba07e
Summary:
These calls make network requests which make our integration tests flaky, this
fix is a followup of D13580144
Reviewed By: StanislavGlebik
Differential Revision: D13580562
fbshipit-source-id: 110563c0360154a86cfc098a05240d7fd023f2c8
