Summary:
For whatever reason, the metadata and vendored sources do not match
the actual upstream sources. I deleted all of third-party/rust/vendor/** and
revendored.
Notable changes:
- libra-crypto and libra-canonical-serialization picked up changes from D25098952 (Nov 2020)
D25986058 updated the source location of z3, but didn't update the sources.
Unfortunately the updated location seems completely unbuildable, so for now
this doesn't update any of z3 until jagill can fix it.
Reviewed By: wangbj
Differential Revision: D26377549
fbshipit-source-id: d30dbaa7502c82c4b7cd64f8c80ab978a5646271
Summary:
Like it says in the title. This adds a crate that provides a combinator that
lets us easily find stalls caused by futures that stay in `poll()` for too
long.
The goal is to make this minimal overhead for whoever is using it: all you need
is to import it + give it a logger. It automatically looks up the line where
it's called and gives it back to you in logs. This uses the `track_caller`
functionality to make this work.
Reviewed By: farnz
Differential Revision: D26250068
fbshipit-source-id: a1458e5adebac7eab6c2de458f679c7215147937
Summary:
Like it says in the title, this splits edenapi_server into 2 crates. Most of
the logic here is pretty straightforward:
- edenapi_service creates a Gotham handler
- edenapi_server instantiates the handler and sets up a socket, tls, etc.
Reviewed By: ahornby
Differential Revision: D26108439
fbshipit-source-id: 6a79e9767ba891265bca11f78eb1a6d3a61ee21f
Summary: Vendor update to smallvec 1.6.1, which is a prerequisite for governor crate.
Reviewed By: jsgf
Differential Revision: D26172450
fbshipit-source-id: 9e87c1ada75b75ac54afd85435cd5422a1c385fa
Summary:
I'd like to be able to use mononoke_api from repo_client, but the hg parts
depend on repo_client, so that's a circular dependency.
This splits out the hg parts of Mononoke API so that places that don't want
them don't have to import them. This is similar to what we did with blobrepo.
Reviewed By: StanislavGlebik
Differential Revision: D26099495
fbshipit-source-id: 73a9c7b5dc95feceb35b5eabccf697e9aa0a27de
Summary:
Like it says in the title, this updates our implementation of Globalrevs to
be a little more relaxed, and allows you to create and move bookmarks as long as
they are ancestors of the "main" Globalrevs bookmark (but NOT to pushrebase to
them later, because we only want to allow ONE globalrevs-publishing bookmark
per repo).
While in there, I also deduplicated how we instantiate pushrebase hooks a
little bit. If anything, this could be better in the pushrebase crate, but
that'd be a circular dependency between pushrebase & bookmarks movement.
Eventually, the callsites should probably be using bookmarks movement anyway,
so leaving pushrebase as the low-level crate and bookmarks movement as the high
level one seems reasonable.
Reviewed By: StanislavGlebik
Differential Revision: D26020274
fbshipit-source-id: 5ff6c1a852800b491a16d16f632462ce9453c89a
Summary:
`blobstore_healer` works by healing blobs in turn, with some level of concurrency. Healing each blob consumes at least `O(blob_size)` or memory, so healing multiple blobs consumes their combined size of memory. Because blob sizes are not distributed uniformly, we cannot just calculate the desired concurrency level once and for all. Prior to this diff, this is what we did and whenever a few multi-gb blobs ended up in the same concurrently-healed batch, the healer OOMed. To help with this problem, this diff starts using dynamic concurrency - it assigns weight to each healed blob and only concurrently heals up to a certain total weight of blobs. This way, we can limit the total amount of memory consumed by our healer.
This solution is not perfect for a variety of reasons:
- if a single blob is larger than the total allowed weight, we'll still let it through. It's better than never healing it, but it means that OOMs are still possible in theory.
- we do not yet know the sizes of all the blobs in the queue. To mitigate that, I took a look at the known sizes distribution and saw that between 0 and 2KB is the most common size range. I defaulted to 1KB size of the unknown blob
Note 1: I had to make `heal_blob` consume it's args instead of borrowing them because `buffered_weight_limited` needs `'static` lifetime for the futures.
Note 2: When using `futures_ext`, I explicitly rename them to `futures_03_ext`, despite the fact that `blobstore_healer` does not depend on the older version. This is because `Cargo.toml` uses the same `[dependencies]` section for the combined dependencies of all the targets in the same `TARGETS` file. As there are other targets that claim the name of `futures_ext` for 0.1 version, I decided that it's easier to just use `_03_` name here than fix in other places. We can always change that of course.
Reviewed By: krallin
Differential Revision: D26106044
fbshipit-source-id: 4931d86d6e85d055ed0eefdd357b9ba6266a1c37
Summary:
Provide a command that measures per-generation sizes, to be run in
Chronos.
Reviewed By: ahornby
Differential Revision: D25981511
fbshipit-source-id: 7d9877dd5d04ac6881b5024e0ba132c234499d5a
Summary:
I plan to use these functions in the megarepotool, so let's move them to a
library that can be used in both.
Reviewed By: krallin
Differential Revision: D26015773
fbshipit-source-id: 0d2d28d86471c417508494883b69fb64e1bea328
Summary:
This diff adds some basic information. Note I have no idea what most of
these crates do so the descriptions are a bit light. I'd appreciate it if people
could tell me what the descriptions should be. Or better yet, just put up a diff
for it after this one goes in.
Reviewed By: lukaspiatkowski
Differential Revision: D26031870
fbshipit-source-id: 496f44882b6ed80a4f3834cf0fa3215ab318c930
Summary:
Lots of generated code in this diff. Only code change was in
`common/rust/cargo_from_buck/lib/cargo_generator.py`.
Path/git-only dependencies (ie `mydep = { path = "../foo/bar" }`) are not
publishable to crates.io. However, we are allowed to specify both a path/git
_and_ a version. When building locally, the path/git is chosen. When publishing,
the version on crates.io is chosen.
See https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#multiple-locations .
Note that I understand that not all autocargo projects are published on crates.io (yet).
The point of this diff is to allow projects to slowly start getting uploaded.
The end goal is autocargo generated `Cargo.toml`s that can be `cargo publish`ed
without further modification.
Reviewed By: lukaspiatkowski
Differential Revision: D26028982
fbshipit-source-id: f7b4c9d4f4dd004727202bd98ab10e201a21e88c
Summary:
Pull Request resolved: https://github.com/facebookincubator/resctl/pull/8074
1. Upgrade `Cursive` to v0.16.3
2. patch `cursive_buffered_backend` with a temp fork before v0.4.2 is released. This is needed to avoid Rust being confused by `Backend` trait from different versions of `cursive_core`
Reviewed By: dschatzberg
Differential Revision: D25987535
fbshipit-source-id: 6826653f9b6d4d6bdba485abf42eb517a853acc1
Summary:
When we tried to update to Tokio 0.2.14, we hit lots of hangs. Those were due
to incompatibilities between Tokio 0.2.14 and Futures 1.29. We fixed some of
the bugs (and others had been fixed and were pending a release), and Futures
1.30 have now been released, which unblocks our update.
This diff updates Tokio accordingly (the previous diff in the stack fixes an
incompatibility).
The underlying motivation here is to ease the transition to Tokio 1.0.
Ultimately we'll be pulling in those changes one or way or another, so let's
get started on this incremental first step.
Reviewed By: farnz
Differential Revision: D25952428
fbshipit-source-id: b753195a1ffb404e0b0975eb7002d6d67ba100c2
Summary:
I'd like to update our connection acceptor to support ALPN in order to
distinguish hgcli traffic from http traffic (the former will have ALPN and the
latter will not).
To do so, I need to parse ALPN, which is a length-prefixed bytestring format.
Let's add support for it in a library. This is what this does.
Reviewed By: johansglock
Differential Revision: D25954536
fbshipit-source-id: 3756dde90906a5f0f36630b866bcb17d03bc94c6
Summary:
This feature is useful for testing time-dependent stuff (e.g. it
allows you to stop/forward time). It's already included in the buck build.
Reviewed By: SkyterX
Differential Revision: D25946732
fbshipit-source-id: 5e7b69967a45e6deaddaac34ba78b42d2f2ad90e
Summary: Add a benchmark tool for bulkops that uses the criterion benchmarking framework. This is so we can measure effect of optimizations later in stack.
Differential Revision: D25804026
fbshipit-source-id: 71b8addf1145c0ecb69d6392b4602172f2b52080
Summary:
This updates Gotham. Under the hood I rebased our fork, you can see the diff
here: P161171514.
The stuff that is relevant is that Gotham got rid of its dependency on
`failure` and now uses `anyhow` instead, and I also added a little bit to our
existing socket data patch by making a few things public so that we can get
access to a few more internals.
Reviewed By: StanislavGlebik
Differential Revision: D25850262
fbshipit-source-id: 25ebf5d63be39e3e93208705d91abc5c61c90453
Summary: Convert all BlobRepoHg methods to new type futures
Reviewed By: StanislavGlebik
Differential Revision: D25471540
fbshipit-source-id: c8e99509d39d0e081d082097cbd9dbfca431637e
Summary:
Like it says in the title, this adds a caching layer around Globalrevs using
our existing `GetOrFillMultipleFromCacheLayers` abstraction.
Note: I've opted to not track dedicated metrics for this (compare to the hg
mapping to see them), since I don't believe we really ever look at them.
I'd like to do a little bit of refactoring of
`GetOrFillMultipleFromCacheLayers` to a) track them without having to ad-hoc
code it, b) convert it 0.3 futures, and c) require less ceremony to call it.
However, I'll do so in another diff.
Reviewed By: StanislavGlebik
Differential Revision: D25334478
fbshipit-source-id: 1385298b8fdf1cd081b6e509c6c3e03b3abbfa5b
Summary:
`ObservabilityContext` is a structure that helps logging facilities within Mononoke to make logging decisions. Specifically, the upcoming `DynamicLoggingDrain` and already existing `MononokeScubaSampleBuilder` will have this structure as a component and decide whether a particular logging statement (slog or scuba) should go ahead.
Internally, `ObservabilityContext` is a wrapper around data received from a [configerator endpoint](https://www.internalfb.com/intern/configerator/edit/?path=scm%2Fmononoke%2Fobservability%2Fobservability_config.cconf).
This diff makes a few unobvious decisions about how this is organized. My goals were:
1. to have production (i.e. reading from configerator), static (i.e. emulating current prod behavior) and test variants of `ObservabilityContext`
1. to avoid having consumers know which of these variants are used
1. to avoid making all consumers of `ObservabilityContext` necessarily generic
1. to avoid using dynamic dispatch of `ObservabilityContext`'s methods
Points 3 and 4 mean that `ObservabilityContext` cannot be a trait. `enum` is a common solution in such cases. However, if `ObservabilityContext` is an `enum`, consumers will know which variant they are using (because `enum` variants are public). So the solution is to use a private enum wrapped in a struct.
Reviewed By: mitrandir77
Differential Revision: D25287759
fbshipit-source-id: da034c71570137e8a8fb7749b1e4ad43be482f66
Summary:
Improve backfill_derived_data logging to make it clear when a chunk is started
and when already-derived changesets have been skipped.
Fix the counters to take into account skiped changesets, and make the time
remaining human-readable.
Reviewed By: StanislavGlebik
Differential Revision: D25099919
fbshipit-source-id: b39d4b99ef0e79d253de2aa0c91f93276b561680
Summary:
Add skeleton manifests to `derived-data verify-manifests` and add a
`skeleton-manifests` subcommand that allows listing and traversing of skeleton
manifests.
Since skeleton manfiests don't contain the content id or file type, they are
valid simply if they contain the same set of files as the manifest we are
comparing against.
To allow testing of these items while backfilling is still in progress, add an
`--if-derived` option to these commands that work even if derivation is
disabled, provided the manifest has already been derived.
Reviewed By: ikostia
Differential Revision: D25064785
fbshipit-source-id: a6f923bfc53262a5b2118917f8fdd3e99407e036
Summary: `lru-disk-cache` depends on an old version of `linked-hash-map` which contains UB in 1.48 (see https://github.com/mozilla/sccache/issues/813). They updated the deps in their repo months ago, but haven't pushed a new version. This diff makes us get `lru-disk-cache` directly from their GitHub instead.
Reviewed By: dtolnay
Differential Revision: D25134582
fbshipit-source-id: 05fd63a76b7095ebeea458645b92a83bbd8c4614
Summary:
We got a few ubns because one bookmark validator in a single region wasn't able
to connect to mysql and was reporting errors.
This diff fixes by separating logical and infra errors
Reviewed By: ikostia
Differential Revision: D25092364
fbshipit-source-id: 93f4be1a7e0467051b7b8d927eef9b4f5cd6a983
Summary:
`manifest/test_utils` contains test utilities that are only used by derived data, and only one
of which relates to manifests. Its name (`test_utils`) is also confusing with `tests/utils`.
Move it to `derived_data_test_utils`, and update it to new futures.
Reviewed By: mitrandir77
Differential Revision: D24787536
fbshipit-source-id: 7a4a735132ccf81e3f75683c7f44c9ada11bc9d7
Summary: As part of the effort to deprecate futures 0.1 in favor of 0.3 I want to create a new futures_ext crate that will contain some of the extensions that are applicable from the futures_01_ext. But first I need to reclame this crate name by renaming the old futures_ext crate. This will also make it easier to track which parts of codebase still use the old futures.
Reviewed By: farnz
Differential Revision: D24725776
fbshipit-source-id: 3574d2a0790f8212f6fad4106655cd41836ff74d
Summary:
The command reads the last SegmentedChangelog that was saved for a repository
and updates head to match a given bookmark (master).
Right now this is just a command that works on one repository. Follow up
changes will look at deployment options and handling multiple repositories.
Reviewed By: krallin
Differential Revision: D24516438
fbshipit-source-id: 8f04f9426c2f2d7748c5363d2dbdf9f3acb79ddd
Summary: This is very old code that once acted to prototype walker-type functionality. As it's dead, delete it.
Reviewed By: ikostia, krallin
Differential Revision: D24591123
fbshipit-source-id: 663108e123d354243c2be4f00819f39d6951db93
Summary:
We want to be able to detect garbage blobs by looking at generation numbers.
Update generation numbers on put, and have a mark command exist to mark blobs as not garbage.
Reviewed By: ahornby
Differential Revision: D23989289
fbshipit-source-id: d96f38649151e3dbd5297cffc262776e74f6cc86
Summary: SQLBlob GC (next diff in stack) will need a ConfigStore in SQLBlob. Make one available to blobstore creation
Reviewed By: krallin
Differential Revision: D24460586
fbshipit-source-id: ea2d5149e0c548844f1fd2a0d241ed0647e137ae
Summary:
a simple command that can be used to pushrebase a commit. Note that this
command is very low level so it skips a lot of checks that we do during the
normal push process (e.g. it doesn't run hooks).
Reviewed By: krallin
Differential Revision: D24534960
fbshipit-source-id: 0f1e27005fa450c86c310d66e3215747ca6a49e2
Summary:
We'll use this command to change the mapping version we use when doing push
redirection.
Reviewed By: ikostia
Differential Revision: D24392308
fbshipit-source-id: 4dab01c0e58a8953a0c6c84c7c166977a6baf00f
Summary:
This diff add new mode of tailing based on derived data graph, it uses same functionality as backfill.
- `tail_batch_iteration` uses `bounded_traversal_dag` directly instead of leveraging `DeriveGraph::derive` so we could warm-up dependencies for each node before calling `DerivedUitls::backfill_batch_dangerous`
Reviewed By: StanislavGlebik
Differential Revision: D24306156
fbshipit-source-id: 006cb6d4df9424cd6501cb4a381b95f096e70551
Summary:
Bookmark filler doesn't make much sense outside of FB. In fact the commit
filler is already in the `facebook/` dir.
D24253307 contains the fbpkg change that has to be landed in-sync with this one.
Reviewed By: lukaspiatkowski
Differential Revision: D24253070
fbshipit-source-id: 52734ae34779801b4cae4882a6d0880586ef505f
Summary:
Mononoke command for running the SegmentedChangelogSeeder for an existing
repository. The result is going to be a new IdMap version in the metadata
store and a new IdDag stored in the the blobstore resulting in a brand new
SegmentedChangelog bundle.
Reviewed By: krallin
Differential Revision: D24096963
fbshipit-source-id: 1eaf78392d66542d9674a99ad0a741f24bc2cb1b
Summary:
This just adds a single fn. I did not come up with a better place/name to put
it, suggestions are welcome. Seems generic enough to belong at the top-level
common location.
I've already needed this twice, so decided to extract. Second callsite will be further in the stack.
Reviewed By: StanislavGlebik
Differential Revision: D24080193
fbshipit-source-id: c3e0646f263562f3eed93f1fdbab9a076729f33c
Summary: Building criterion in opt mode degradated enormously after moving to 0.3.3, pin it to 0.3.1 for now until we figure out what is the problem.
Reviewed By: ljw1004
Differential Revision: D24046885
fbshipit-source-id: 6373eb06b5f47061cc02597bf82f574511fbec43
Summary:
This diff makes blobstore healer to use MyAdmin to get replication lag for a DB shard and removes "laggable" interface for connections.
The old "laggable" API worked this way: we maintained potential connections to each possible region, then tried to query replica status on all of them. If there was no replica hosts in some of the regions, we just wanted to ignore it by handling a specific error type.
This is legacy and makes the logic more complicated. We want for the new code to use Myadmin instead.
Reviewed By: krallin
Differential Revision: D23767442
fbshipit-source-id: 9f85f07bd318ad020d203d2bcd1c8898061f7572
Summary: Just as the previous diff, but this time it removes reverse_mover
Reviewed By: ikostia
Differential Revision: D23879509
fbshipit-source-id: ed111ca2d106120229c4facc0bb2435913c27966
