Summary:
Now that the `hg_external_sync` jobs are gone we can delete the code
in Mononoke that behaves differently when a sync job connects.
Reviewed By: StanislavGlebik
Differential Revision: D27500506
fbshipit-source-id: 443fb54577833dbf44ece6ae90a5f25ffed38cd5
Summary:
I'd like to just get rid of that library since it's one more place where we
specify the Tokio version and that's a little annoying with the Tokio 1.x
update. Besides, this library is largely obsoleted by `#[fbinit::test]` and
`#[tokio::test]`.
Reviewed By: farnz
Differential Revision: D27619147
fbshipit-source-id: 4a316b81d882ea83c43bed05e873cabd2100b758
Summary: Use the test factory for existing hooks and mapping tests.
Reviewed By: StanislavGlebik
Differential Revision: D27169436
fbshipit-source-id: f814e462bb2244e117c83bd355042016f5bfde8e
Summary:
Currently we can only limit which users are allowed to move a bookmark by a
regex. We also want to allow specifying a hipster group.
Reviewed By: krallin
Differential Revision: D27156690
fbshipit-source-id: 99a5678a82f4c34ed2e57625361ba7cdb08ed839
Summary:
If a filename passes this hook it should be checkoutable on windows.
Unfortunately it's not the case now - we're missing the check for invalid
characters. Although this list might be redundant and some of chars might be
already banned by other hook let's make this hook a place for all
windows-specific filename checks.
Reviewed By: farnz
Differential Revision: D26979418
fbshipit-source-id: f1fa685b9e7e5413d8030d18bc083458b6a148e1
Summary:
BonsaiChangesets are rarely mutated, and their maps are stored in sorted order,
so we can use `SortedVectorMap` to load them more efficiently.
In the cases where mutable maps of filechanges are needed, we can use `BTreeMap`
during the mutation and then convert them to `SortedVectorMap` to store them.
Reviewed By: mitrandir77
Differential Revision: D25615279
fbshipit-source-id: 796219c1130df5cb025952bb61002e8d2ae898f4
Summary:
AsyncVfs provides async vfs interface.
It will be used in the native checkout instead of current use case that spawns blocking tokio tasks for VFS action
Reviewed By: quark-zju
Differential Revision: D26801250
fbshipit-source-id: bb26c4fc8acac82f4b55bb3f2f3964a6d0b64014
Summary: For the OWNERS files in `secure` and `core` we need to check that all the groups and users mentioned in the OWNERS are in the secure owners group.
Reviewed By: StanislavGlebik
Differential Revision: D26761772
fbshipit-source-id: 02ad2bc45c82792e51702cd4d8d092557c76c015
Summary:
This diffs add a layer of indirection between fbinit and tokio, thus allowing
us to use fbinit with tokio 0.2 or tokio 1.x.
The way this works is that you specify the Tokio you want by adding it as an
extra dependency alongside `fbinit` in your `TARGETS` (before this, you had to
always include `tokio-02`).
If you use `fbinit-tokio`, then `#[fbinit::main]` and `#[fbinit::test]` get you
a Tokio 1.x runtime, whereas if you use `fbinit-tokio-02`, you get a Tokio 0.2
runtime.
This diff is big, because it needs to change all the TARGETS that reference
this in the same diff that introduces the mechanism. I also didn't produce it
by hand.
Instead, I scripted the transformation using this script: P242773846
I then ran it using:
```
{ hg grep -l "fbinit::test"; hg grep -l "fbinit::main" } | \
sort | \
uniq | \
xargs ~/codemod/codemod.py \
&& yes | arc lint \
&& common/rust/cargo_from_buck/bin/autocargo
```
Finally, I grabbed the files returned by `hg grep`, then fed them to:
```
arc lint-rust --paths-from ~/files2 --apply-patches --take RUSTFIXDEPS
```
(I had to modify the file list a bit: notably I removed stuff from scripts/ because
some of that causes Buck to crash when running lint-rust, and I also had to add
fbcode/ as a prefix everywhere).
Reviewed By: mitrandir77
Differential Revision: D26754757
fbshipit-source-id: 326b1c4efc9a57ea89db9b1d390677bcd2ab985e
Summary:
This diff rollouts V2 of autocargo in an atomic way so there are quite a few things done here.
Arc lint support:
V1 used to be part of the default fbsource `arc lint` engine, but since V2 calls buck it must live in a separate lint engine. So this diff:
- Adds running `autocargo` as part of `arc lint-rust`
Mergedriver update:
- Mergedriver used in resolving conflicts on commits is now pointing to V2
- It handles files in `public_autocargo/` directories in addition to the ones containig generation preamble
Including regeneration results of running `common/rust/cargo_from_buck/bin/autocargo`. All the differences are accounted for:
- Some sections and attributes are removed as they can be autodiscovered by Cargo (like `lib.path = "src/lib.rs"` or empty [lib] section)
- "readme" attribute is properly defined as relative to Cargo.toml location rather than as hardcoded string
- "unittest = false" on a Buck rule propagates as "test = false; doctest = false" to Cargo
- "rusqlite" is not special-cased anymore, so the "budled" feature will have to be enabled using custom configuration if required by the project (for rust-shed in order to not break windows builds a default feature section was added)
- Files generated from thrift_library rules that do not support "rust" language are removed
- Custom .bzl rules that create rust artifacts (like `rust_python_extension`) are no longer ignored
Others:
- Changed `bin/cargo-autocargo` to be a wrapper for calling V2 via `cargo autocargo`
- Updated following files to use V2:
- `common/rust/tools/reindeer/version-bump`
- `remote_execution/rust/setup.sh`
- Removed few files from V1 that would otherwise interfere with V2 automatic regeneration/linting/testing
Reviewed By: zertosh
Differential Revision: D26728789
fbshipit-source-id: d1454e7ce658a2d3194704f8d77b12d688ec3e64
Summary:
For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present.
Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is.
The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed.
Reviewed By: ikostia
Differential Revision: D26618363
fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
Summary:
The on demand update code we have is the most basic logic that we could have.
The main problem is that it has long and redundant write locks. This change
reduces the write lock strictly to the section that has to update the in memory
IdDag.
Updating the Dag has 3 phases:
* loading the data that is required for the update;
* updating the IdMap;
* updating the IdDag;
The Dag can function well for serving requests as long as the commits involved
have been built so we want to have easy read access to both the IdMap and the
IdDag. The IdMap is a very simple structure and because it's described as an
Arc<dyn IdMap> we push the update locking logic to the storage. The IdDag is a
complicated structure that we ask to update itself. Those functions take
mutable references. Updating the storage of the iddag to hide the complexities
of locking is more difficult. We deal with the IdDag directly by wrapping it in
a RwLock. The RwLock allows for easy read access which we expect to be the
predominant access pattern.
Updates to the dag are not completely stable so racing updates can have
conflicting results. In case of conflics one of the update processes would have
to restart. It's easier to reason about the process if we just allow one
"thread" to start an update process. The update process is locked by a sync
mutex. The "threads" that fail the race to update are asked to wait until the
ongoing update is complete. The waiters will poll on a shared future that
tracks the ongoing dag update. After the update is complete the waiters will go
back to checking if the data they have is available in the dag. It is possible
that the dag is updated in between determining that the an update is needed and
acquiring the ongoing_update lock. This is fine because the update building
process checks the state of dag before the dag and updates only what is
necessary if necessary.
Reviewed By: krallin
Differential Revision: D26508430
fbshipit-source-id: cd3bceed7e0ffb00aee64433816b5a23c0508d3c
Summary:
The earlier diffs in this stack have removed all our dependencies on the Tokio
0.1 runtime environment (so, basically, `tokio-executor` and `tokio-timer`), so
we don't need this anymore.
We do still have some deps on `tokio-io`, but this is just traits + helpers,
so this doesn't actually prevent us from removing the 0.1 runtime!
Note that we still have a few transitive dependencies on Tokio 0.1:
- async-unit uses tokio-compat
- hg depends on tokio-compat too, and we depend on it in tests
This isn't the end of the world though, we can live with that :)
Reviewed By: ahornby
Differential Revision: D26544410
fbshipit-source-id: 24789be2402c3f48220dcaad110e8246ef02ecd8
Summary:
The changes (and fixes) needed were:
- Ignore rules that are not rust_library or thrift_library (previously only ignore rust_bindgen_library, so that binary and test dependencies were incorrectly added to Cargo.toml)
- Thrift package name to match escaping logic of `tools/build_defs/fbcode_macros/build_defs/lib/thrift/rust.bzl`
- Rearrange some attributes, like features, authors, edition etc.
- Authors to use " instead of '
- Features to be sorted
- Sort all dependencies as one instead of grouping third party and fbcode dependencies together
- Manually format certain entries from third-party/rust/Cargo.toml, since V2 formats third party dependency entries and V1 just takes them as is.
Reviewed By: zertosh
Differential Revision: D26544150
fbshipit-source-id: 19d98985bd6c3ac901ad40cff38ee1ced547e8eb
Summary:
OWNERS hook sometimes needs to know the most recent changes made to the OWNERS files. So the user, who was last to change the OWNERS file, can change it again.
This diffs extends `FileContentManager` trait to have a new method `recent_changes`. It derives unode manifest for the given bookmark, finds given paths and gets linknodes. For the linknodes it derives `ChangesetInfo` which is a lightweight version of the bonsai changesets, they contain all the metadata, but don't have a set file changes.
We use unodes here instead of hg manifests, because hg manifests don't have correct linknodes.
Reviewed By: StanislavGlebik
Differential Revision: D26468603
fbshipit-source-id: 78dd0a4e51bee6228de1296eafe7ffa393df2fbe
Summary:
Autocargo V2 will use a more structured format for autocargo field
with the help of `cargo_toml` crate it will be easy to deserialize and handle
it.
Also the "include" field is apparently obsolete as it is used for cargo-publish (see https://doc.rust-lang.org/cargo/reference/manifest.html#the-exclude-and-include-fields). From what I know this might be often wrong, especially if someone tries to publish a package from fbcode, then the private facebook folders might be shipped. Lets just not set it and in the new system one will be able to set it explicitly via autocargo parameter on a rule.
Reviewed By: ahornby
Differential Revision: D26339606
fbshipit-source-id: 510a01a4dd80b3efe58a14553b752009d516d651
Summary:
johansglock pointed out that Hyper is affected by CVE-2021-21299. Let's update
to a fixed version.
Reviewed By: farnz
Differential Revision: D26313854
fbshipit-source-id: 4db04d3044fb9f22a037bda0a88a5314f62f9dfc
Summary:
This diff introduces abstractions for the OWNERS files and the rules. For now rules don't have any acl checkers and just store the parsed user and group names as a lists of strings.
Now we find and fetch all the possible owners files using `FileContentFetcher` new API.
Reviewed By: StanislavGlebik
Differential Revision: D25379329
fbshipit-source-id: 0933ef4e5e960102b46af9ed2f3e0b12d9761000
Summary: The `FileContentFetcher` trait with the new API like `find_content` or `file_changes` looks more like a "manager" rather than a "fetcher".
Reviewed By: StanislavGlebik
Differential Revision: D26040535
fbshipit-source-id: 20ab3b0730e4a2b1cbe2991c3c13a9d8a8ec8f35
Summary: Loading an Acl Checker is expensive. Don't do it given we don't use it.
Reviewed By: HarveyHunt
Differential Revision: D26337699
fbshipit-source-id: bc48ac44b3301db5bd03a5d2006b92912b7e6fd0
Summary:
This diff adds method to FileContentFetcher to get a file changes between two changesets using hg manifests. The method returns a list of file changes, where added/changed paths contain content ids.
The API is needed for Owners files hook.
Reviewed By: StanislavGlebik
Differential Revision: D26040536
fbshipit-source-id: 816194a494f0c77e8ea9aef729fd6161eed904ff
Summary:
Owners file hook requires access to the files in master branch.
This diff extends `FileContentFetcher` to have API for finding files in the master commit. The methods are implemented for the blobrepo. They derive unode manifest for the master bookmark and search given file paths in the manifest, returning list of file content ids.
Reviewed By: StanislavGlebik
Differential Revision: D24936674
fbshipit-source-id: c63931c073b6148972bc8e81126f7c4786db920a
Summary:
Just a minor version update. I'd like to add a patch on top of this for a PR
that hasn't been merged yet, but updating to the underlying released version
first will make the diff clearer.
Reviewed By: ahornby
Differential Revision: D26047997
fbshipit-source-id: 91856f645ec3aaaf4fbf256a23c7e8d4db0f6b37
Summary:
Lots of generated code in this diff. Only code change was in
`common/rust/cargo_from_buck/lib/cargo_generator.py`.
Path/git-only dependencies (ie `mydep = { path = "../foo/bar" }`) are not
publishable to crates.io. However, we are allowed to specify both a path/git
_and_ a version. When building locally, the path/git is chosen. When publishing,
the version on crates.io is chosen.
See https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#multiple-locations .
Note that I understand that not all autocargo projects are published on crates.io (yet).
The point of this diff is to allow projects to slowly start getting uploaded.
The end goal is autocargo generated `Cargo.toml`s that can be `cargo publish`ed
without further modification.
Reviewed By: lukaspiatkowski
Differential Revision: D26028982
fbshipit-source-id: f7b4c9d4f4dd004727202bd98ab10e201a21e88c
Summary:
When we tried to update to Tokio 0.2.14, we hit lots of hangs. Those were due
to incompatibilities between Tokio 0.2.14 and Futures 1.29. We fixed some of
the bugs (and others had been fixed and were pending a release), and Futures
1.30 have now been released, which unblocks our update.
This diff updates Tokio accordingly (the previous diff in the stack fixes an
incompatibility).
The underlying motivation here is to ease the transition to Tokio 1.0.
Ultimately we'll be pulling in those changes one or way or another, so let's
get started on this incremental first step.
Reviewed By: farnz
Differential Revision: D25952428
fbshipit-source-id: b753195a1ffb404e0b0975eb7002d6d67ba100c2
Summary:
This feature is useful for testing time-dependent stuff (e.g. it
allows you to stop/forward time). It's already included in the buck build.
Reviewed By: SkyterX
Differential Revision: D25946732
fbshipit-source-id: 5e7b69967a45e6deaddaac34ba78b42d2f2ad90e
Summary:
This is useful data to see how much we are over the limit. I noticed
the missing data while resolving
https://fb.workplace.com/groups/scm/permalink/3450652774984318/ as the oncall.
Reviewed By: StanislavGlebik
Differential Revision: D25414427
fbshipit-source-id: ec4bbca9c21a4bf0e675ec1cd82e4e703cd88631
Summary:
This diff prepares the Mononoke codebase for composition-based extendability of
`ScubaSampleBuilder`. Specifically, in the near future I will add:
- new methods for verbose scuba logging
- new data field (`ObservabilityContext`) to check if verbose logging should
be enabled or disabled
The higher-level goal here is to be able to enable/disable verbose Scuba
logging (either overall or for certain slices of logs, like for a certain
session id) in real time, without restarting Mononoke. To do so, I plan to
expose the aforementioned verbose logging methods, which will run a check
against the stored `ObservabilityContext` and make a decision of whether the
logging is enabled or not. `ObservabilityContext` will of course hide
implementation details from the renamed `ScubaSampleBuilderExt`, and just provide a yes/no
answer based on the current config and sample fields.
At the moment this should be a completely harmless change.
Reviewed By: krallin
Differential Revision: D25211089
fbshipit-source-id: ea03dda82fadb7fc91a2433e12e220582ede5fb8
Summary:
There seem to be legitimate cases when we want to have both bypass mechanisms available: via the commit message and via the pushvar. The need to be able to use a commit message arises from the fact that engineers rarely have access to push CLI, so they cannot use pushvars. Still, pushvars are useful if we want to push an existing commit without any changes to its message.
NB. Since I wanted to make `HookBypass` a multi-option, I had to change it from `enum` to `struct`. If I left fields `pub` in that struct (the way it's commonly done in `metaconfig`, I would've allowed `HookBypass` instances with both options `None`. This is not a big deal, but is a bit ugly as we already store `Option<HookBypass>` instead, so it's better if `Some(hook_bypass)` symbolizes that there indeed is some way to bypass the hook. So I opted in for constructors, getters and private fields.
Reviewed By: StanislavGlebik
Differential Revision: D25000578
fbshipit-source-id: ad13076c9690ee8d45fc4a93e32a6026ff5bdd09
Summary:
Like it says in the title. This required quite a lot of changes at callsites,
as you'd expect.
Reviewed By: StanislavGlebik
Differential Revision: D24731299
fbshipit-source-id: e58447e88dcc3ba1ab3c951f87f7042e2b03eb2c
Summary:
This updates the metadata APIs in the filestore to futures 0.3 & async / await.
This changes the external API of the filestore, so there's quite a bit of churn
outside of that module.
Reviewed By: markbt
Differential Revision: D24727255
fbshipit-source-id: 59833f185abd6ab9c609c6bcc22ca88ada6f1b42
Summary:
The root cause for S199754 was a file named "con.rs" was checked in onto the
repo. Since this is a reserved filename on Windows, this broke all Windows
users having it in their sparse profiles.
The rules for reserved names are defined as such by Microsoft:
"CON, PRN, AUX, NUL, COM1, COM2, COM3, COM4, COM5, COM6, COM7, COM8, COM9,
LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, and LPT9. Also avoid these
names followed immediately by an extension; for example, NUL.txt is not
recommended. For more information, see Namespaces."
Of course, since the filesystem is case insensitive, these can have any casing.
Reviewed By: krallin
Differential Revision: D24453528
fbshipit-source-id: 389f15e2b1a88e3c1e8721fb7868616acabebc64
Summary: Members of `scm` hipster group will be able to push to mononoke bypassing hooks when `BYPASS_ALL_HOOKS` pushvar is passed.
Reviewed By: krallin
Differential Revision: D24477468
fbshipit-source-id: ac910bf27e5510e1975c4a7cd0bfeff5216da70e
Summary: It will be much more convenient for end user to know why certain limit is enforced.
Reviewed By: krallin
Differential Revision: D24364473
fbshipit-source-id: ad86384f7f7a246bc3735028339cfd54a1d0254a
Summary:
Until we have the same standards for the native and push-redirected pushes,
these need to be automatically bypassed.
Reviewed By: krallin
Differential Revision: D24357372
fbshipit-source-id: f85459145f6a5217c07445d7017f3b11ed1284a7
Summary:
Likely `MPath` constructor got more clever since this was written. In any case
I am just happy that it builds.
Reviewed By: farnz
Differential Revision: D24278551
fbshipit-source-id: 2378a43bd01b1826b7f81d294a035053fba5322f
Summary: For the purposes of megarepo, we need to treat certain paths differently depending on whether the push is native to a large repo, or is push-redirected from a small repo. Let's add a separate patterns list to the `deny_files` hook. Of course, we can also add a list, specific for push-redirected pushes, but it seems unneeded atm.
Reviewed By: StanislavGlebik
Differential Revision: D24278409
fbshipit-source-id: 9fd815940bb656ceac6ab234f3a0647a5c57db06
Summary:
This change adds some wiring to allow the hooks to treat native vs push-redirected pushes differently. This is needed because `deny-files` needs to block native pushes to `.ovrsource-rest` and `arvr-legacy`, while it needs to allow push-redirection into these directories.
The plan further is to change the actual hook body for `deny_files` to have different handling of the two cases.
Reviewed By: StanislavGlebik
Differential Revision: D24257454
fbshipit-source-id: 2f5931149115210aeeeebb3294a6512effd36350
Summary: `clippy` often complains about the use of `.len() != 0`, `.len() > 0` or `.len() == 0`and proposes to use `.is_empty()` instead. This diff does that across Mononoke.
Reviewed By: aslpavel
Differential Revision: D24099427
fbshipit-source-id: 1bba2f958485b7efb3f41bf3eae820879c92b0e5
Summary:
Interngraph token is already stored within keychain service. We should make use of that.
I'll need to remove related config option in convigerator in a separate diff.
Reviewed By: krallin
Differential Revision: D24015463
fbshipit-source-id: 9e8246e2cc252f0c42669140de7b50410a15709c
