Summary:
Vendoring this patch: https://github.com/curl/curl/pull/7737 to curl-sys rust crate. On windows the hg client is using curl that's bundled with sys-curl. I need this patch to have unix domain sockets working in hg client on windows.
I had to manually vendor https://raw.githubusercontent.com/mzr/curl/57e7ec4dbe4dd2831de51f2644879387d2ea7b44/docs/INSTALL because reindeer didn't do it. IDK why.
oss-eden-{darwin,linux,windows}-getdeps fail with:
```
FAILED: eden/scm/lib/backingstore/CMakeFiles/rust_backingstore.cargo eden/scm/lib/backingstore/debug/libbackingstore.a eden/scm/lib/backingstore/release/libbackingstore.a
cd /data/sandcastle/temp/fbcode_builder_getdeps/shipit/eden/eden/scm/lib/backingstore && /data/sandcastle/temp/fbcode_builder_getdeps/installed/cmake-hQhVzQT-WzFKTeqXjLxo5lLi8IG4_MjX2-YRqptCUVs/bin/cmake -E remove -f /data/sandcastle/temp/fbcode_builder_getdeps/shipit/eden/eden/scm/lib/backingstore/Cargo.lock && /data/sandcastle/temp/fbcode_builder_getdeps/installed/cmake-hQhVzQT-WzFKTeqXjLxo5lLi8IG4_MjX2-YRqptCUVs/bin/cmake -E env CARGO_TARGET_DIR=/data/sandcastle/temp/fbcode_builder_getdeps/build/eden/eden/scm/lib/backingstore CARGO_HOME=/data/sandcastle/temp/fbcode_builder_getdeps/build/eden/_cargo_home cargo build --release -p backingstore --features fb
Blocking waiting for file lock on package cache
Blocking waiting for file lock on package cache
error: failed to calculate checksum of: /data/sandcastle/boxes/eden-trunk-hg-fbcode-fbsource/third-party/rust/vendor/curl-sys-0.4.45+curl-7.78.0/curl/docs/INSTALL
Caused by:
failed to open file `/data/sandcastle/boxes/eden-trunk-hg-fbcode-fbsource/third-party/rust/vendor/curl-sys-0.4.45+curl-7.78.0/curl/docs/INSTALL`
Caused by:
No such file or directory (os error 2)
```
Not idea how to fix this. Seems related to the fact that reindeer didn't vendor docs/INSTALL.
# EDIT:
# It might been caused by some bug in hg. now it's fine
# The failure in oss-eden-linux-getdeps looks unrelated (something with rocksdb)
Reviewed By: krallin
Differential Revision: D31370778
fbshipit-source-id: a1245f8cb6b58f5765e34c95dfd78325a8e6e457
Summary:
This diff adds snapshot tests for most eden api wire types, while at the same time making the testing code much smaller, including tests for "wire" and "serialize" roundtrips.
## Context:
The previous diff had added an easy way to add snapshot tests. This stack aims to simplify the wire protocol code needed to create/modify an endpoint. A good thing to do before that is to add snapshot tests to all wire types, so that if we change them in a refactor, we're confident they still work exactly the same. This will also be useful when a type is changed in the future.
## How this makes tests easier
- In order to create snapshot tests, we need example objects to test with. Luckily we already use a framework for generating example objects (quickcheck::Arbitrary), so the idea here is to use that to make snapshot tests as automatic as possible.
- At the same time, the "wire" and "serialize" roundtrip tests (which also used Arbitrary), can also be made more automatic.
Now, using a simple helper, `auto_wire_tests!(WireObjectName)`, it is possible to derive all three types of tests automatically. This makes the current code smaller, and safer as we now have the additional safety provided by snapshot tests.
## Observations
- Not all wire types had tests implemented for them (I assume because it was too much work doing so, and might have done that myself in the past), I only moved the ones that already had. I'll do another pass and add remaining objects on a following diff.
- There are a couple actual non-refactor changes. I'll add comments explaining those.
- quickcheck crate is using quite an old version. I tried updating but it snowballed into something much more complicated, so I kept using the old version. We'll need to get to it at some point, though.
Reviewed By: markbt
Differential Revision: D31019233
fbshipit-source-id: 30c4a90848d0a5dcaffb89b9a0cd1cebfe4ace55
Summary: Need this for cargo check/rust-analyzer to work.
Reviewed By: guswynn
Differential Revision: D31319911
fbshipit-source-id: ebd3fa72d8fc3667391a2067f95cab9e5f53301f
Summary: I'm parsing some deeply-nested JSON, and it's running into the limit. This feature enables a potential footgun, but even with the feature enabled you have to add code to reach for said footgun.
Reviewed By: jsgf
Differential Revision: D31284743
fbshipit-source-id: 00ea5d7d7db8bdeb878d48fe390831f39e007409
Summary:
This updates the following crates to their latest versions:
- `rand`: 0.7 => 0.8
- `quickcheck`: 0.9 => 1.0
Both crates introduced some breaking changes, so affected clients had to be fixed accordingly. Most changes are rather mechanical and shouldn't change the existing logic. In addition, a few buggy property tests were uncovered, presumably due to `quicktest` becoming smarter with its choice of inputs in the newer version, and the fixes are included in this diff.
Reviewed By: yancouto
Differential Revision: D31115820
fbshipit-source-id: 60a61dfac3236fd93cd4f03b86506654d81d330f
Summary: Share the megarepo api object as it contains repo configs which are quite large
Reviewed By: StanislavGlebik
Differential Revision: D31144245
fbshipit-source-id: bc87bac8d51d255d3f0a2d6dfbacf97ec89a565d
Summary:
We update bookmarks and insert sync mappings in separate transactions. Let's
adjust for that (see comment for details)
Reviewed By: StanislavGlebik
Differential Revision: D31113203
fbshipit-source-id: 95264bac0dfffee3fef82d475ccf7c4a62b81f24
Summary:
Those two commands allow us to easily make another job pick-up unfinished
request and finish a request with error.
aborting will actually make the worker stop working on the requests the next
time worker tries to update the heartbeat.
Reviewed By: StanislavGlebik
Differential Revision: D30965078
fbshipit-source-id: 5b683ea09c7dd74ff903416f2a1d393f9b55c588
Summary: ACE will have to run in situations where Chef has not run, but we'll need to be able to reliably write to the auth logs so Blackbird can properly build detections. So we need these crates so we can build the somewhat foolproof solution to ensure ACE logs all executions.
Reviewed By: farnz
Differential Revision: D31066559
fbshipit-source-id: 9fa3b5778cd2602bdeaac90a9daa758b117babfe
Summary:
Backport: https://github.com/briansmith/ring/pull/1334
This will allow us to unpin Rust compiler to 1.53.0 and update to 1.55.0.
Reviewed By: xavierd
Differential Revision: D31039024
fbshipit-source-id: f6a9c918e836d93d03c34c77c12bbe63cf7cbe09
Summary:
this relies on local changes to make it so cargo metadata ACTUALLY finds this
binary: https://github.com/tokio-rs/console/pull/146 is where I try to upstream
it
Reviewed By: jsgf
Differential Revision: D30944630
fbshipit-source-id: 5d34a32a042f83eff7e7ae7445e23badf10fffe3
Summary: For the time being we don't have checksums in saved states. As a temporary workaround add the ability to derive the checksum from the naming table.
Differential Revision: D30967637
fbshipit-source-id: 4ac34d988d08c9af6f08f7ce46206f756cf1cf0c
Summary:
This is a very basic commands that uses debug-printing to display all the
request details. In the future we might want to make it more ellaborate but
as-it-is it works.
Reviewed By: StanislavGlebik
Differential Revision: D30965076
fbshipit-source-id: 561c64597b94359843e575550be0ae6f39fad7bf
Summary:
This debug command will allow the user to see and interact with currently
running async requests.
Reviewed By: StanislavGlebik
Differential Revision: D30965077
fbshipit-source-id: 259f1af0eb6ade4a34f6004c7b1ad63cd5f0bc9f
Summary:
This allows the mononoke_api user to choose whether the skiplists
should be used to spped up the ancestry checks or not.
The skiplists crate is already prepared for the situation where skiplist
entries are missing and traverses the graph then.
Reviewed By: yancouto
Differential Revision: D30958909
fbshipit-source-id: 7773487b78ac6641fa2a427c55f679b49f99ac8d
Summary:
Allow the mononoke_api user to choose whether they want
oprerations to be sped up using WBC or not.
Reviewed By: yancouto
Differential Revision: D30958908
fbshipit-source-id: 038cf77735e7c655f6801d714762e316b6817df5
Summary: We use it as an unique key for the detector
Reviewed By: ginfung
Differential Revision: D30703470
fbshipit-source-id: cb8e7dae5dc4192402530b2cfe564b86aa23c7c8
Summary:
`rust_include_srcs` is supported on `thrift_library` as a way of including other Rust code in the generated crate, generally used to implement other traits on the generated types.
Adding support for this in autocargo by copying these files into the output dir and making sure their option is specified to the thrift compiler
Reviewed By: ahornby
Differential Revision: D30789835
fbshipit-source-id: 325cb59fdf85324dccfff20a559802c11816769f
Summary:
Add impls for Layer for Box/Arc<L: Layer> and <dyn Layer>. Also a pile of other
updates in git which haven't been published to crates.io yet, including proper
level filtering of trace events being fed into log.
Reviewed By: dtolnay
Differential Revision: D30829927
fbshipit-source-id: c01c9369222df2af663e8f8bf59ea78ee12f7866
Summary:
Bump all the versions on crates.io to highest to make migration to github
versions in next diff work.
Reviewed By: dtolnay
Differential Revision: D30829928
fbshipit-source-id: 09567c26f275b3b1806bf8fd05417e91f04ba2ef
Summary:
Currently there are two things preventing us from running add_sync_target
on existing target:
* already existing bookmark
* already existing config
Both need to be deleted to create new target. This diff removes the second
one to simplify code and make it easier to recreate the target (it's easy to
forget about manually removing the config as they otherwise don't need
human interventions).
Reviewed By: StanislavGlebik
Differential Revision: D30767613
fbshipit-source-id: f951c0e1ef9bde69d805dc911331fcdb220123f2
Summary:
Like it says in the title, this updates us to use Daemonize 0.5, though from
Github and not Crates.io, because it hasn't been released to the latter yet.
The main motivation here is to pull in
https://github.com/knsd/daemonize/pull/39 to avoid leaking PID files to
children of the daemon.
This required some changes in `hphp/hack/src/facebook/hh_decl` and `xplat/rust/mobium` since the way to
run code after daemonization has changed (and became more flexible).
Reviewed By: ndmitchell
Differential Revision: D30694946
fbshipit-source-id: d99768febe449d7a079feec78ab8826d0e29f1ef
Summary:
Manual component version update
Bump Schedule: https://www.internalfb.com/intern/msdk/bump/?schedule_fbid=342556550408072
Package: https://www.internalfb.com/intern/msdk/package/181247287328949/
Oncall Team: rust_foundation
NOTE: This build is expected to expire at 2022/09/01 09:14AM PDT
---------
New project source changes since last bump based on D30663071 (08e362a355e0a64a503f5073f57f927394696b8c at 2021/08/31 03:47AM -05):
| 2021/08/31 04:41AM -05 | generatedunixname89002005294178 | D30665384 | [MSDK] Update autocargo component on FBS:master |
| 2021/08/31 07:14PM PDT | kavoor | D30681642 | [autocargo] Make cxx-build match version of cxx |
| 2021/09/01 04:05PM BST | krallin | D30698095 | autocargo: include generated comment in OSS manifests |
---------
build-break (bot commits are not reviewed by a human)
Reviewed By: farnz
Differential Revision: D30717040
fbshipit-source-id: 2c1d09f0d51b6ff2e2636496cf22bcf781f22889
Summary:
The mockall crate's `automock` attribute previously created nondeterministic output, which leads to frequent random "Found possibly newer version of crate" failures in Buck builds that involve cache.
The affected trait in Conveyor is:
https://www.internalfb.com/code/fbsource/[4753807291f7275a061d67cead04ea12e7b38ae2]/fbcode/conveyor/common/just_knobs/src/lib.rs?lines=13-23
which has a method with two lifetime parameters. Mockall's generated code shuffled them in random order due to emitting the lifetimes in HashSet order. The generated code would randomly contain one of these two types:
`Box<dyn for<'b, 'a> FnMut(&str, Option<&'a str>, Option<&'b str>) -> Result<bool> + Send>`
`Box<dyn for<'a, 'b> FnMut(&str, Option<&'a str>, Option<&'b str>) -> Result<bool> + Send>`
Reviewed By: jsgf
Differential Revision: D30656936
fbshipit-source-id: c1a251774333d7a4001a7492c1995efd84ff22e5
Summary:
add_sync_target might need to derive a lot of data, and it takes a long time to
do it. We don't have any resumability, so if it fails for any reason, then we'd
need to start over.
For now let's just retry a few times so that we don't have to start over
because of flakiness
Reviewed By: mitrandir77
Differential Revision: D30511785
fbshipit-source-id: 1a9c5e62db366022ad487ed108dd41b1dea4caa2
Summary:
These functions were used heavily, and used old futures.
That meant a lot of uselessly copied stuff, and harder interop.
I also took the opportunity to pass `CoreContext` around as a reference, and did some BlobRepo refactoring to use attribute traits where possible.
Reviewed By: StanislavGlebik
Differential Revision: D30368261
fbshipit-source-id: 2e63677601fafa3c2e3d9d3340df0a5f31a19a11
Summary:
The diff is giant, but it's just a one-line change to add the
nested-values feature to slog, we just have a whole bunch of projects dependent
on slog.
Reviewed By: dtolnay
Differential Revision: D30351289
fbshipit-source-id: b6c1c896b06cbdf23b1f92c0aac9a97aa116085d
Summary:
In preparation for the derived data manager, ensure that derived data
mappings do not require a `BlobRepo` reference.
The main use for this was to log to scuba. This functionality is extracted out
to the new `BonsaiDerivedMappingContainer`, which now contains just enough
information to be able to log to scuba.
Reviewed By: mitrandir77
Differential Revision: D30135447
fbshipit-source-id: 1daa468a87f297adc531cb214dda3fa7fe9b15da
Summary:
We have mover only for files, and it doesn't quite work for directories - at
the very least directory can be None (i.e. root of the repo).
In the next diffs we'll start recording files and directories renames during
megarepo operations, so let's DirectoryMultiMover as a preparation for that.
Reviewed By: mitrandir77
Differential Revision: D30338444
fbshipit-source-id: 4fed5f50397a7d3d8b77f23552921d515a684604
Summary:
AOSP megarepo wants to create release branches from existing branches, and then update configs to follow only release-ready code.
Provide the primitive they need to do this, which takes an existing commit and config, and creates a new config that tracks the same sources. The `change_target_config` method can then be used to shift from mainline to release branch
Reviewed By: StanislavGlebik
Differential Revision: D30280537
fbshipit-source-id: 43dac24451cf66daa1cd825ada8f685957cc33c1
Summary:
This diff adds some data to BonsaiChangeset that tells whether it is a snapshot or not.
For now, this marks every changeset as not being a snapshot. The next diff will add validation to snapshots, some tests, and mark the current `snapshot createremote` command as uploading snapshots.
Reviewed By: markbt
Differential Revision: D30158530
fbshipit-source-id: 9835450ac44e39ce8d653938f3a629f081247d2f
Summary:
Autocargo only allows 1 rust-library per Cargo.toml, but right now we have 3
per Thrift library so that doesn't work:
https://www.internalfb.com/intern/sandcastle/log/?instance_id=27021598231105145&step_id=27021602582167211&step_index=13&name=Run%20config
There's little benefit in Autocargo-ifying those rules anyway since they're of
use to Thrift servers and this doesn't work at all in our OSS builds, so let's
just see if we can just noop them. That'll make the crate not exist at all as a
dep, but even considering that it exists only to link to a C++ library that
Autocargo doesn'tk now how to build anyway, that seems OK?
drop-conflicts
Reviewed By: markbt
Differential Revision: D30304720
fbshipit-source-id: 047524985b2dadab8610267c05e3a1b3770e84e6
Summary: This adds types to FileChange thrift and rust structs to deal with additional possible snapshot states, that is, untracked and missing files. Conflicted stuff not added yet.
Reviewed By: StanislavGlebik
Differential Revision: D30103162
fbshipit-source-id: 59faa9e4af8dca907b1ec410b8af74985d85b837
Summary:
for now this changes:
```
struct FileChange {
...stuff
}
fn f(x: Option<FileChange>)
```
to
```
struct TrackedFileChange {
...stuff
}
enum FileChange {
TrackedChange(TrackedFileChange),
Deleted,
}
fn f(x: FileChange)
```
This makes it much clearer that `None` actually means the file was deleted. It will also be useful as in the next diff I will add more stuff inside FileChange (for untracked changes), and this refactor will make it easy.
(The refactor from using `Option` to putting it all inside the enum isn't really necessary, but IMO it looks much clearer, so I did it.)
Reviewed By: StanislavGlebik
Differential Revision: D30103454
fbshipit-source-id: afd2f29dc96baf9f3d069ad69bb3555387cff604
Summary: Previous diffs switched all our usage from chashmap to dashmap as dashmap upstream is more responsive. Now remove chashmap from the cargo vendoring.
Reviewed By: dtolnay
Differential Revision: D30046522
fbshipit-source-id: 111ef9375bd8095f8b7c95752ecbc1988fb0438d
Summary:
In integration tests, we want to be able to run through the megarepo processing, and then check that configs have persisted correctly, so that we can start async workers after sending a config change down, and see the change be picked up.
Make it possible
Reviewed By: StanislavGlebik
Differential Revision: D30012106
fbshipit-source-id: f944165e7b93451180a78d8287db8a59d71bbe13
Summary:
The use of dyn traits of the Thrift-generated server traits was emitting future compatibility warnings with recent versions of rustc, due to a fixed soundness hole in the trait object system:
```
error: the trait `x_account_aggregator_if::server::XAccountAggregator` cannot be made into an object
|
= this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!
note: for a trait to be "object safe" it needs to allow building a vtable to allow the call to be resolvable dynamically; for more information visit <https://doc.rust-lang.org/reference/items/traits.html#object-safety>
```
This diff pulls in https://github.com/dtolnay/async-trait/releases/tag/0.1.51 which results in the Thrift-generated server traits no longer hitting the problematic pattern.
Reviewed By: zertosh
Differential Revision: D29979939
fbshipit-source-id: 3e6e976181bfcf35ed453ae681baeb76a634ddda
Summary:
Just as with D29874802 and D29848377, let's make sure if the same
sync_changeset request was sent again then we would return the same result.
Reviewed By: mojsarn
Differential Revision: D29876414
fbshipit-source-id: 91c3bd38983809da8ce246f44066204df667bb12
Summary:
# Goal of the stack
There goal of this stack is to make megarepo api safer to use. In particular, we want to achieve:
1) If the same request is executed a few times, then it won't cause corrupt repo in any way (i.e. it won't create commits that client didn't intend to create and it won't move bookmarks to unpredictable places)
2) If request finished successfully, but we failed to send the success to the client, then repeating the same request would finish successfully.
Achieving #1 is necessary because async_requests_worker might execute a few requests at the same time (though this should be rare). Achieveing #2 is necessary because if we fail to send successful response to the client (e.g. because of network issues), we want client to retry and get this successful response back, so that client can continue with their next request.
In order to achieve #1 we make all bookmark move conditional i.e. we move a bookmark only if current location of the bookmark is at the place where client expects it. This should help achieve goal #1, because even if we have two requests executing at the same time, only one of them will successfully move a bookmark.
However once we achieve #1 we have a problem with #2 - if a request was successful, but we failed to send a successful reply back to the client then client will retry the request, and it will fail, because a bookmark is already at the new location (because previous request was successful), but client expects it to be at the old location (because client doesn't know that the request was succesful). To fix this issue before executing the request we check if this request was already successful, and we do it heuristically by checking request parameters and verifying the commit remapping state. This doesn't protect against malicious clients, but it should protect from issue #2 described above.
So the whole stack of diffs is the following:
1) take a method from megarepo api
2) implement a diff that makes bookmark moves conditional
3) Fix the problem #2 by checking if a previous request was successful or not
# This diff
Now that we have target_location in sync_changeset() method,
let's move bookmark in sync_changeset conditionally, just as in D29874803 (5afc48a292).
This would prevent race conditions from happening when the same sync_changeset
method is executing twice.
Reviewed By: krallin
Differential Revision: D29876413
fbshipit-source-id: c076e14171c6615fba2cedf4524d442bd25f83ab
Summary:
# Goal of the stack
There goal of this stack is to make megarepo api safer to use. In particular, we want to achieve:
1) If the same request is executed a few times, then it won't cause corrupt repo in any way (i.e. it won't create commits that client didn't intend to create and it won't move bookmarks to unpredictable places)
2) If request finished successfully, but we failed to send the success to the client, then repeating the same request would finish successfully.
Achieving #1 is necessary because async_requests_worker might execute a few requests at the same time (though this should be rare). Achieveing #2 is necessary because if we fail to send successful response to the client (e.g. because of network issues), we want client to retry and get this successful response back, so that client can continue with their next request.
In order to achieve #1 we make all bookmark move conditional i.e. we move a bookmark only if current location of the bookmark is at the place where client expects it. This should help achieve goal #1, because even if we have two requests executing at the same time, only one of them will successfully move a bookmark.
However once we achieve #1 we have a problem with #2 - if a request was successful, but we failed to send a successful reply back to the client then client will retry the request, and it will fail, because a bookmark is already at the new location (because previous request was successful), but client expects it to be at the old location (because client doesn't know that the request was succesful). To fix this issue before executing the request we check if this request was already successful, and we do it heuristically by checking request parameters and verifying the commit remapping state. This doesn't protect against malicious clients, but it should protect from issue #2 described above.
So the whole stack of diffs is the following:
1) take a method from megarepo api
2) implement a diff that makes bookmark moves conditional
3) Fix the problem #2 by checking if a previous request was successful or not
# This diff
We already have it for change_target_config, and it's useful to prevent races
and inconsistencies. That's especially important given that our async request
worker might run a few identical sync_changeset methods at the same time, and
target_location can help process this situation correctly.
Let's add target_location to sync_changeset, and while there I also updated the
comment for these fields in other methods. The comment said
```
// This operation will succeed only if the
// `target`'s bookmark is still at the same location
// when this operation tries to advance it
```
This is not always the case - operation might succeed if the same operation has been
re-sent twice, see previous diffs for more explanationmotivation.
Reviewed By: krallin
Differential Revision: D29875242
fbshipit-source-id: c14b2148548abde984c3cb5cc62d04f920240657
