Summary: If some blob is missing it becomes very hard to debug what is missing. This diff extends `LoadableError::Missing` with string argument.
Reviewed By: krallin
Differential Revision: D17132049
fbshipit-source-id: df1f60b1f739c8a837d1b1798edbaca58b1587f9
Summary: It is often cumbersome to add work with `find_entries` when we only need entry for single path. This function simplifies this case by adding helper function which only works with single path `find_entry`.
Reviewed By: StanislavGlebik
Differential Revision: D17115042
fbshipit-source-id: 0c4ae0d0b61994a2f04a119ec78a0de0394fdf0a
Summary:
This updates our bookmark prefix matching to operate properly even in the
presence of characters at the top of the range of valid characters (right now,
that's ASCII, and that character is DEL, but that's applicable regardless of
the range).
Before this diff, our matching just added a DEL at the end of the prefix.
That's not entirely correct. For example, if your prefix is foo, then you
should match foo[DEL]. Our existing implementation wouldn't match it.
Likewise, if the prefix is the empty string, then [DEL]foo should match, but
we'd exclude it.
This diff fixes that. This problem isn't really meaningful to fix for our day
to day (except to feel good about correctness, I suppose), but it is important
so that we can update to a newer version of the AsciiString crate, which now
generates DEL characters in quickcheck and thus broke our tests.
Instead of appending a DEL, this updated implementation updates the last
character in the prefix to the be the next character in ASCII. This actually
raises 3 possibilities:
- There is no last character (because the prefix is empty)
- There is no next character after the last character (because it's a DEL)
- It works
In the 1st case, we just need to return a range that explicitly matches
everything. In the second, we either go back one more character and find the
next one there, or eventually return a unbounded range (so, if you give us
[DEL][DEL] as a prefix, we'll match everything from there upwards). In the last
case, we can return a regular Range. To make those cases work, I implemented
RangeBounds on a enum that represents those possibilities.
Reviewed By: aslpavel, HarveyHunt
Differential Revision: D17114013
fbshipit-source-id: 1ef57aef5585bb115df23c5499c7dc7f8d73e599
Summary: Add --drain-only option to blobstore_healer so we can drain a queue without healing when required.
Reviewed By: ikostia
Differential Revision: D17111436
fbshipit-source-id: e2fba4914e9699a4836933d0f35def015cd3dadc
Summary: Log the region with max replication lag as well as the lag size
Reviewed By: StanislavGlebik
Differential Revision: D17097266
fbshipit-source-id: c449167b09dbd2cdce7d65bfc956e7ccf59b2415
Summary:
blobstore_healer performance improvements
* speed up delete by using bulk delete
* speed up rep lag check by only doing it once if lag within bounds
* log some stats from each healer chunk run to help diagnose issues
* dont log the full storage config on each log line ( this was making it hard to read the logs )
Reviewed By: StanislavGlebik
Differential Revision: D17093809
fbshipit-source-id: 34299ad1835c5548b9502ad9d5a3e18eff1ab95b
Summary:
Derived data generation is rather slow at the moment - up to 3 mins. I've added
a few printlns that showed that the problem is in the sequential writes.
This diff makes it so that writes to the blobstore happen in parallel.
Reviewed By: aslpavel
Differential Revision: D17093614
fbshipit-source-id: 771ccddfdb742e44a56dbd5be935559447c97298
Summary:
First, checked the following to ensure that all existing `.rlsconfig`
files inherited their `rustfmt` setting from `fbcode/.rlsconfig`:
```
fbcode$ fbgf -s -l .rlsconfig | xargs rg rustfmt
.rlsconfig
5: "rustfmt": "$BUCKROOT/common/rust/tools/common/rustfmt_wrapper.sh",
experimental/ljw/r6/.rlsconfig
4: "rustfmt": "$BUCKROOT/common/rust/tools/common/rustfmt_wrapper.sh",
experimental/ljw/r4/.rlsconfig
4: "rustfmt": "$BUCKROOT/common/rust/tools/common/rustfmt_wrapper.sh",
```
Effectively, this means that all uses of `rustfmt` via our internal
Rust extension for VS Code/Nuclide should go through this script.
Though to make things complicated, when an external `rustfmt` is specified
to `rls`, it performs this step where it takes a complete in-memory config
and serializes it to a temp file, using it as the value of `rustfmt`'s
`--config-path` option.
Unfortunately for us, that includes a config value named `required_version`:
https://github.com/rust-lang/rustfmt/blob/master/Configurations.md#required_version
It defaults to `CARGO_PKG_VERSION`, so for the current `rls` in tp2, this
ends up being `1.2.2` even though we want to use it with an external `rustfmt`
whose version is `1.3.3`. We can workaround this by specifying
`required_version = "1.3.3"` in our own `rustfmt.toml` so that the value in
the serialized config matches the version with which it will be used.
Reviewed By: dtolnay
Differential Revision: D17089540
fbshipit-source-id: 082ceef0f983e7cece6373603a0f318a7bc2ab2f
Summary:
We have slow requests in apiserver, and we'd like to optimize them. Logging how
long derived data generation takes can help us prioritize what to optimize
next.
Note that currently we need to log for hg changeset generation and for derived
data separately. This is a temporary state until we move all hg changeset
generation onto derived data.
Reviewed By: aslpavel, HarveyHunt
Differential Revision: D17090591
fbshipit-source-id: 20e58b75e49c7ea2b50444170d7aabed1efc95ff
Summary:
Utility to backfill derived data:
- currently only support `unodes`
Reviewed By: krallin
Differential Revision: D16827897
fbshipit-source-id: aa0bb8ae4fe895233e3f11775b3432184afb37f9
Summary: Switch all implementation of `find_in_*_manifest` to use `ManifestOps::find_entries`. In later diffs I will remove all methods associated with manifest lookup from blobrepo
Reviewed By: StanislavGlebik
Differential Revision: D17004608
fbshipit-source-id: 1e7eedb8fbb74cb29373cdf1ca73361ab785a8a4
Summary:
This is a part of an effort to move mercurial related stuff into mercurial crate:
- `BlobManifest|HgBlobEntry` -> `mercurial/types/blobs`
- helper `Id` type is no longer required and `BlobManifeset` can be loaded with `HgManifesetId::load`
- `BlobManifest` implements `HgManifest` and typed `Manifest` traits
Reviewed By: StanislavGlebik
Differential Revision: D16984745
fbshipit-source-id: 33d4eb9c33137258956917328efa1f5ec6737ee9
Summary:
Previously we had separate load shedding configs for quicksand and
non-quicksand. They were hard to manage because we had to duplicate each load
shedding option for quicksand and non-quicksand. More importantly we had to
duplicate that in the code, and that was error prone.
Now we have a quicksand_multiplier instead which is applied to every
loadshedding option if request is coming from quicksand. let's use it
Reviewed By: krallin
Differential Revision: D16986262
fbshipit-source-id: 46a2ed9eb6e591d8c8a70f8d57efb1cb7836db78
Summary:
Add two helpful commands:
1) Fetch the phase of a commit
2) List all public commits
Reviewed By: ikostia
Differential Revision: D16830576
fbshipit-source-id: 03f503cb30a7f150ea383d62fb71913dd2b93e6e
Summary:
This is prerequisite step before moving hg blob `fetch|upload` functionality to mercurial crate.
- Use `Arc<dyn Blobstore>` instead of `RepoBlobstore`
- do not use `BlobRepo` in `UploadHg*` and `fetch_*` structures and functions
Reviewed By: ikostia
Differential Revision: D16963058
fbshipit-source-id: be36599cb430a342348cc077885b2f32eaa4ee47
Summary: Just a small cleanup before the next diff
Reviewed By: HarveyHunt
Differential Revision: D16986263
fbshipit-source-id: 85f064ab5b7dc1bfdf0596b9ca1c001851e967db
Summary:
Find entries now support prefixes as well as paths:
- selector can be anything that is convertable to `PathOrPrefix`
- returns `Option<MPath>` instead of `MPath`
Reviewed By: StanislavGlebik, farnz
Differential Revision: D16937401
fbshipit-source-id: 24cce9719d52ce3cbc493575c2a6db5ebf121315
Summary:
RedactedBlobstore is a part of `RepoBlobstore` which is the primary blobstore used in mononoke, it is cloned basically everywhere. This change make it clonable without allocations.
- redacted hash table was cloned on every clone of `RedactedBlostore`
- scuba_builder also contain hash table which was cloned on every clone or `RedactedBlobstore`
Reviewed By: krallin
Differential Revision: D16961607
fbshipit-source-id: 59fe56f0395b0184418ed8b709be8ecdefb9f4ce
Summary:
There are a collection of bookmarks that should only be moved by automation. If they're
moved by users, SEVs can happen (S184864).
Configure the restrict_users hook to disallow moves of the following bookmarks:
fbandroid/stable
fbcode/stable
fbcode/warm
fbobjc/stable
fbsource/stable
I collected this list by running the following on a hg server:
[svcscm@hg009.cln2 /data/scm/fbsource]$ hg config | grep automation
hooks.automationbookmarks=fbandroid/stable\nfbcode/stable\nfbcode/warm\nfbobjc/stable\nfbsource/stable
The usernames came from the disable-nonff.py hook on hg servers.
Reviewed By: suitingtseng, StanislavGlebik
Differential Revision: D16960784
fbshipit-source-id: 9cf63b40492431bf34fb0574738794b0190703c6
Summary:
If an error happens in CreateChangeset::create when we are committing a stack,
the parent ready channel immediately signals, and we end up with an error
that's very unhelpful: "Canceled". This happens because signal_parent_ready
gets dropped.
This patch updates the code to try and report what caused the parent to fail.
We can't clone errors, so all we show is a Debug representation of the error.
In practice, this causes the failure signal to be sent _after_ the parent data
future fails, so we actually see the parent error directly, which is just as
nice.
Reviewed By: farnz
Differential Revision: D16961035
fbshipit-source-id: 8cf3e78ecf785c471425e81f22a67a54d8bc3aeb
Summary:
The high level goal is to test new unodes functionality. In particular, we want
to test `list_directory()` functionality.
There's a problem though - `list_directory()` returns Nodehash and filetype for
each entry it returns. That's cheap to do with mercurial manifests but not
cheap with unodes:
1) Mercurial manifests store filetype (i.e. whether it's a file,
symlink or executable) in manifests. But we don't store it in unode manifests
(it's stored in a separate file unode entry).
2) Unodes don't have mercurial hashes at all.
However there's currently a single user of `list_directory()` call - this is
scmquery - and it just discards both of the fields - https://fburl.com/8vr5hibo.
So I decided to introduce a separate `list_directory_unodes()` method that
returns a list of directory entries without any additional info. The plan is to
use it in scmquery for repos that already has unodes backfilled. That's
intended to be short-term - ideally we replace current thrift interface with
something more modern or at the very least replace `list_directory` with
`list_directory_unodes()` completely.
Reviewed By: farnz
Differential Revision: D16918232
fbshipit-source-id: f05185be8eecbf0852c7ddb0a482d4a5a63f8b53
Summary:
This updates the `mononoke_admin` filenodes command to allow showing filenodes
by path + ID in addition to revision + path. This makes it possible to locate
the linknode given a filenode ID but no revision.
This updates the filenodes CLI command a little bit (so if you want the old
command you need to add `by-revision`), but this should be fine since this is
a command that a human will normally be calling.
Reviewed By: farnz
Differential Revision: D16936628
fbshipit-source-id: 1079ad580b235076739c92d0c1f3c9bfc80db4c1
Summary: This is needed for commit syncing between repos (in particular, stack beginning with D16896918).
Reviewed By: farnz
Differential Revision: D16919646
fbshipit-source-id: 37e09417bab033427479cfd8373e1e4a415a92f5
Summary: This adds a command that allows importing a set of LFS blobs into Mononoke by streaming them out of another storage location (e.g. Dewey).
Reviewed By: HarveyHunt
Differential Revision: D16917806
fbshipit-source-id: 4917d56e11a187c89e00c23a32c6e791b351f8ef
Summary:
Current pushrebase starts by mapping your `HgChangeset`s to `BonsaiChangeset`s. Cross-repo sync will generate bonsai form directly, then pushrebase it.
Reuse the existing pushrebase machinery - this also puts into a good place to start removing HgChangeset knowledge from pushrebase.
Reviewed By: StanislavGlebik
Differential Revision: D16915565
fbshipit-source-id: f6a1e77453eece047c1f3586ce659f9c88cc81e1
Summary: When we sync commits, we will need to be able to cross-reference the commits between the source repo and the target repo. Make it easy to track this mapping.
Reviewed By: StanislavGlebik
Differential Revision: D16896918
fbshipit-source-id: 3738cf2067291f16ce9b62509763ac526d1d35c2
Summary:
We needed this hack because in derive_unode_manifest crate we needed to implement Loadable trait (previously defined in
blobstore crate) for ManifestUnodeId which was defined in mononoke_types crate.
Rust doesn't let us do that - see
https://stackoverflow.com/questions/35160995/implement-foreign-trait-for-foreign-type.
So we used this new Id type, which made it awkward to use. This diff fixes it.
Reviewed By: krallin
Differential Revision: D16885044
fbshipit-source-id: 329b09bb3b671a59824d1d2597a3907bebb82642
Summary:
In order to remove struct Id hack in the next diff we need to implement
Manifest trait for ManifestUnode struct. However, there's a problem:
ManifestUnode::TreeId type is ManifestUnodeId which implements
MononokeId trait. And all MononokeId types automatically implement
Loadable<Value=Option<MononokeId>>. But Manifest trait requires TreeId that
implement Loadable<Value=Self> - note that there's no Option<...>, and so
ManifestUnodeId Loadable implementation doesn't match implementation required
by Manifest trait.
There are a few ways to fix it (see one of them in D16885043), and after some
discussion we decided that the best way to proceed is to add a special error
LoadableError that might be `Missing` meaning that id we are trying to fetch
doesn't exist
Reviewed By: aslpavel
Differential Revision: D16915387
fbshipit-source-id: df57ff4a7ab4ddbff5812ce5929a742d949726af
Summary:
We had copy-pasted implementation for Loadable/Storable in ContentMetadataId
Let's remove it
Reviewed By: krallin
Differential Revision: D16914637
fbshipit-source-id: cead9e64dcddd8a0b5b51faff13b80b350904f57
Summary:
We need this to accomodate `blobimport`'s needs of using (opening(D16914875)
and creating(this)) unredacted repos.
Note: this is nearly identical to D16914875.
Reviewed By: StanislavGlebik
Differential Revision: D16915557
fbshipit-source-id: 6b522e6fcee4582870df27cf66e8ad55e91235fe
Summary:
Currently blobimport fails if it tries to import a redacted blob. We should
only enforce redaction in the scenarios where end users could see the data.
Internal imports should be fine to proceed.
As a separate note, problems like this would probably go away when redaction
would be moved to the `FileStore` level.
Reviewed By: krallin
Differential Revision: D16914875
fbshipit-source-id: e531a6248f66b47de87c7aedcbc928bb7ad4ebc9
Summary:
We do not want to use censorship/blacklisting, as those names are ambiguous
and redaction has exactly the meaning we need.
Reviewed By: quark-zju
Differential Revision: D16890423
fbshipit-source-id: 1a6dcc61ca3fe77b7a938a37fb1ec9250ae9e15a
Summary:
This is a mechanical part of rename, does not change any commit messages in
tests, does not change the scuba table name/config setting. Those are more
complex.
Reviewed By: krallin
Differential Revision: D16890120
fbshipit-source-id: 966c0066f5e959631995a1abcc7123549f7495b6
Summary:
mononoke_types doesn't look like a good crate for it. Let's move to blobstore
crate instead
Reviewed By: krallin
Differential Revision: D16914142
fbshipit-source-id: 57602c5a644adc87beea0cbc92beeb0d901d5873
Summary:
We now have two Manifest and two Entry traits and it makes it very confusing.
The old Manifest trait (and related EmptyManifest struct and Entry trait) are
mercurial-specific, while new Manifest trait is generic over all kinds of
manifests (unodes, mercurial etc).
In ideal state we should have only new Manifest trait, but it'd require quite a
big refactoring. In short term let's at least rename Manifest to HgManifest and
Entry to HgEntry to make distinction clearer.
Reviewed By: krallin
Differential Revision: D16886032
fbshipit-source-id: 80e3999fae3e89347448579caf7aafec56705500
Summary:
This is the first diff in the stack that removes ugly hack `struct Id` that we
used in derived data (see more details about that later in the stack).
This diff reverses the dependency between blobstore crate and mononoke_types
crate. Previously blobstore depended on mononoke_types, and blobstore also had
Loadable/Storable traits. That means that we can't implement Loadable/Storable
traits for types from mononoke_types - we can't even import them without
creating cyclic dependency between crates! And we'll use Loadable in
mononoke_types later in the stack.
This diff reverses the dependency and moves Loadable/Storable in mononoke_types
Reviewed By: krallin
Differential Revision: D16884649
fbshipit-source-id: 20753f3e62980be7bb6a04ab9d19980bc459b40e
Summary:
The blob repo creation code was accidentally not calling open_filenodes when
creating a Filenodes DB without caching. This means that if you tried to
connect to filenodes with `--skip-caching` set, you wouldn't go to the sharded
version of this DB.
(this had been the case historically "no caching" used to mean "use SQLite",
but unfortunately I missed this when I added support for no caching + not
SQLite in D15941644 and left the non-sharding implementation in there)
Fortunately, the impact of this should be fairly minimal since we effectively
run everything that matters with caching turned on.
While debugging this, I got tripped up by the `filenodes-shards` CLI argument.
It's unused, so this diff removes it.
Reviewed By: HarveyHunt
Differential Revision: D16894641
fbshipit-source-id: 151c1f5df76b2e5d77214143974912100427817a
Summary:
There are a couple of functions that take a reference to a blobstore and then
immediately clone it, in order to pass ownership to another function.
Modify these functions to take ownership.
Reviewed By: farnz
Differential Revision: D16859228
fbshipit-source-id: fd7b0ae5af2262601db2201aa510d3099f37ced5
Summary: Proper duration types can help catch more bugs at compile time than passing around usize. This moves the Rust Configerator client's timeout argument from usize to a proper duration type.
Reviewed By: dtolnay
Differential Revision: D16854216
fbshipit-source-id: a067c13a56a1dc8e5bad5c9f6243774fd9e0b462
