Summary:
I ran into a tricky case while deriving FastlogBatch for empty commits:
* If a commit is empty then new unode manifest is not created [1]
* But FastlogBatch's derived data logic was still creating a new FastlogBatch.
This batch would point not to the commit that created the unode, but to the
latest empty commit.
This is obviously not a huge deal because it affects only root entry only for
empty commits. But it can be confusing and tricky to investigate, so I think
it's better to fix it.
[1] There is one exception - empty commit with no parents
Reviewed By: krallin
Differential Revision: D17160654
fbshipit-source-id: 949d35860efe2f3df6ec1c11b4c6c4d077982c95
Summary:
This diff adds handling of "overflows" i.e. making sure that each FastlogBatch
doesn't store more than 60 entries.
The idea is that FastlogBatch has "latest" entries and "previous_batches". New
entry is prepended to "latest", but if "latest" has more than 10 entries then
new FastlogBatch is uploaded and prepended to "previous_batches". If
"previous_batches" has more than 5 entries than the oldest entries is removed
from the batch.
Note that merges are still not handled in this diff, this functionality will be added in the next
diffs
Reviewed By: krallin
Differential Revision: D17160506
fbshipit-source-id: 956bc5320667c6c5511d2a567740a4f6ebd8ac1b
Summary:
This diff adds batching of linear file or directory history (i.e. no merges).
Note that it still doesn't handle "overflows" i.e. FastlogBatch is not yet
compressed if it has too many entries. This will be added in the next diffs
Reviewed By: krallin
Differential Revision: D17152698
fbshipit-source-id: 05706444bcdcdb5ed0fd453c5c0684f8911603c6
Summary:
In order to answer file history requests history we need to avoid doing serial
fetches from a blobstore.
This diff is a first attempt in implementing it. The general idea is to store precalculated history in the blobstore.
Note that at the moment it covers only the very basic case - file history for a commit with no parents, so there's quite a few "TODOs" and "unimplemented!()" in the codebase. This functionality will be extended in the next diffs
Reviewed By: krallin
Differential Revision: D17070901
fbshipit-source-id: 8150afb2509fcd2a428d2369bab58468ab774d72
Summary:
We were generating hgchangesets in list_directory_unodes() method, however this
is not always necessary - if a client sends a bookmark then we can fetch bonsai
changesets and generate unodes directly bypassing hg changeset generation stage.
Same goes for is_ancestor method.
Reviewed By: krallin
Differential Revision: D17146609
fbshipit-source-id: 9613e28f8bacbce5d8de94a6ab88b152d19b0a08
Summary:
Change the settings we use for repo_client histogram stats to better
align with actual performance numbers. This prevents clumping in the histogram's
buckets that cause inaccurate percentile estimates.
Folly's histogram allows you to specify a maximum value, a minimum value and
also a bucket width. The range min to max is split into buckets according to
the bucket size. There are 2 extra buckets created as catch alls - one for
anything less than min and another for anything greater than max. These buckets
are used to calculate metrics such as percentiles.
Reviewed By: krallin
Differential Revision: D17132608
fbshipit-source-id: ead89c49dbce797e8b93f0dac24fe8895a2623b6
Summary: Adds `fb303` back_fille_derived data when it runs in tailing mode
Reviewed By: krallin
Differential Revision: D17130955
fbshipit-source-id: d16fd2a729227ae3450c1c152a21a8ae8a01d7fb
Summary: If some blob is missing it becomes very hard to debug what is missing. This diff extends `LoadableError::Missing` with string argument.
Reviewed By: krallin
Differential Revision: D17132049
fbshipit-source-id: df1f60b1f739c8a837d1b1798edbaca58b1587f9
Summary: It is often cumbersome to add work with `find_entries` when we only need entry for single path. This function simplifies this case by adding helper function which only works with single path `find_entry`.
Reviewed By: StanislavGlebik
Differential Revision: D17115042
fbshipit-source-id: 0c4ae0d0b61994a2f04a119ec78a0de0394fdf0a
Summary:
This updates our bookmark prefix matching to operate properly even in the
presence of characters at the top of the range of valid characters (right now,
that's ASCII, and that character is DEL, but that's applicable regardless of
the range).
Before this diff, our matching just added a DEL at the end of the prefix.
That's not entirely correct. For example, if your prefix is foo, then you
should match foo[DEL]. Our existing implementation wouldn't match it.
Likewise, if the prefix is the empty string, then [DEL]foo should match, but
we'd exclude it.
This diff fixes that. This problem isn't really meaningful to fix for our day
to day (except to feel good about correctness, I suppose), but it is important
so that we can update to a newer version of the AsciiString crate, which now
generates DEL characters in quickcheck and thus broke our tests.
Instead of appending a DEL, this updated implementation updates the last
character in the prefix to the be the next character in ASCII. This actually
raises 3 possibilities:
- There is no last character (because the prefix is empty)
- There is no next character after the last character (because it's a DEL)
- It works
In the 1st case, we just need to return a range that explicitly matches
everything. In the second, we either go back one more character and find the
next one there, or eventually return a unbounded range (so, if you give us
[DEL][DEL] as a prefix, we'll match everything from there upwards). In the last
case, we can return a regular Range. To make those cases work, I implemented
RangeBounds on a enum that represents those possibilities.
Reviewed By: aslpavel, HarveyHunt
Differential Revision: D17114013
fbshipit-source-id: 1ef57aef5585bb115df23c5499c7dc7f8d73e599
Summary: Add --drain-only option to blobstore_healer so we can drain a queue without healing when required.
Reviewed By: ikostia
Differential Revision: D17111436
fbshipit-source-id: e2fba4914e9699a4836933d0f35def015cd3dadc
Summary: Log the region with max replication lag as well as the lag size
Reviewed By: StanislavGlebik
Differential Revision: D17097266
fbshipit-source-id: c449167b09dbd2cdce7d65bfc956e7ccf59b2415
Summary:
blobstore_healer performance improvements
* speed up delete by using bulk delete
* speed up rep lag check by only doing it once if lag within bounds
* log some stats from each healer chunk run to help diagnose issues
* dont log the full storage config on each log line ( this was making it hard to read the logs )
Reviewed By: StanislavGlebik
Differential Revision: D17093809
fbshipit-source-id: 34299ad1835c5548b9502ad9d5a3e18eff1ab95b
Summary:
Derived data generation is rather slow at the moment - up to 3 mins. I've added
a few printlns that showed that the problem is in the sequential writes.
This diff makes it so that writes to the blobstore happen in parallel.
Reviewed By: aslpavel
Differential Revision: D17093614
fbshipit-source-id: 771ccddfdb742e44a56dbd5be935559447c97298
Summary:
First, checked the following to ensure that all existing `.rlsconfig`
files inherited their `rustfmt` setting from `fbcode/.rlsconfig`:
```
fbcode$ fbgf -s -l .rlsconfig | xargs rg rustfmt
.rlsconfig
5: "rustfmt": "$BUCKROOT/common/rust/tools/common/rustfmt_wrapper.sh",
experimental/ljw/r6/.rlsconfig
4: "rustfmt": "$BUCKROOT/common/rust/tools/common/rustfmt_wrapper.sh",
experimental/ljw/r4/.rlsconfig
4: "rustfmt": "$BUCKROOT/common/rust/tools/common/rustfmt_wrapper.sh",
```
Effectively, this means that all uses of `rustfmt` via our internal
Rust extension for VS Code/Nuclide should go through this script.
Though to make things complicated, when an external `rustfmt` is specified
to `rls`, it performs this step where it takes a complete in-memory config
and serializes it to a temp file, using it as the value of `rustfmt`'s
`--config-path` option.
Unfortunately for us, that includes a config value named `required_version`:
https://github.com/rust-lang/rustfmt/blob/master/Configurations.md#required_version
It defaults to `CARGO_PKG_VERSION`, so for the current `rls` in tp2, this
ends up being `1.2.2` even though we want to use it with an external `rustfmt`
whose version is `1.3.3`. We can workaround this by specifying
`required_version = "1.3.3"` in our own `rustfmt.toml` so that the value in
the serialized config matches the version with which it will be used.
Reviewed By: dtolnay
Differential Revision: D17089540
fbshipit-source-id: 082ceef0f983e7cece6373603a0f318a7bc2ab2f
Summary:
We have slow requests in apiserver, and we'd like to optimize them. Logging how
long derived data generation takes can help us prioritize what to optimize
next.
Note that currently we need to log for hg changeset generation and for derived
data separately. This is a temporary state until we move all hg changeset
generation onto derived data.
Reviewed By: aslpavel, HarveyHunt
Differential Revision: D17090591
fbshipit-source-id: 20e58b75e49c7ea2b50444170d7aabed1efc95ff
Summary:
Utility to backfill derived data:
- currently only support `unodes`
Reviewed By: krallin
Differential Revision: D16827897
fbshipit-source-id: aa0bb8ae4fe895233e3f11775b3432184afb37f9
Summary: Switch all implementation of `find_in_*_manifest` to use `ManifestOps::find_entries`. In later diffs I will remove all methods associated with manifest lookup from blobrepo
Reviewed By: StanislavGlebik
Differential Revision: D17004608
fbshipit-source-id: 1e7eedb8fbb74cb29373cdf1ca73361ab785a8a4
Summary:
This is a part of an effort to move mercurial related stuff into mercurial crate:
- `BlobManifest|HgBlobEntry` -> `mercurial/types/blobs`
- helper `Id` type is no longer required and `BlobManifeset` can be loaded with `HgManifesetId::load`
- `BlobManifest` implements `HgManifest` and typed `Manifest` traits
Reviewed By: StanislavGlebik
Differential Revision: D16984745
fbshipit-source-id: 33d4eb9c33137258956917328efa1f5ec6737ee9
Summary:
Previously we had separate load shedding configs for quicksand and
non-quicksand. They were hard to manage because we had to duplicate each load
shedding option for quicksand and non-quicksand. More importantly we had to
duplicate that in the code, and that was error prone.
Now we have a quicksand_multiplier instead which is applied to every
loadshedding option if request is coming from quicksand. let's use it
Reviewed By: krallin
Differential Revision: D16986262
fbshipit-source-id: 46a2ed9eb6e591d8c8a70f8d57efb1cb7836db78
Summary:
Add two helpful commands:
1) Fetch the phase of a commit
2) List all public commits
Reviewed By: ikostia
Differential Revision: D16830576
fbshipit-source-id: 03f503cb30a7f150ea383d62fb71913dd2b93e6e
Summary:
This is prerequisite step before moving hg blob `fetch|upload` functionality to mercurial crate.
- Use `Arc<dyn Blobstore>` instead of `RepoBlobstore`
- do not use `BlobRepo` in `UploadHg*` and `fetch_*` structures and functions
Reviewed By: ikostia
Differential Revision: D16963058
fbshipit-source-id: be36599cb430a342348cc077885b2f32eaa4ee47
Summary: Just a small cleanup before the next diff
Reviewed By: HarveyHunt
Differential Revision: D16986263
fbshipit-source-id: 85f064ab5b7dc1bfdf0596b9ca1c001851e967db
Summary:
Find entries now support prefixes as well as paths:
- selector can be anything that is convertable to `PathOrPrefix`
- returns `Option<MPath>` instead of `MPath`
Reviewed By: StanislavGlebik, farnz
Differential Revision: D16937401
fbshipit-source-id: 24cce9719d52ce3cbc493575c2a6db5ebf121315
Summary:
RedactedBlobstore is a part of `RepoBlobstore` which is the primary blobstore used in mononoke, it is cloned basically everywhere. This change make it clonable without allocations.
- redacted hash table was cloned on every clone of `RedactedBlostore`
- scuba_builder also contain hash table which was cloned on every clone or `RedactedBlobstore`
Reviewed By: krallin
Differential Revision: D16961607
fbshipit-source-id: 59fe56f0395b0184418ed8b709be8ecdefb9f4ce
Summary:
There are a collection of bookmarks that should only be moved by automation. If they're
moved by users, SEVs can happen (S184864).
Configure the restrict_users hook to disallow moves of the following bookmarks:
fbandroid/stable
fbcode/stable
fbcode/warm
fbobjc/stable
fbsource/stable
I collected this list by running the following on a hg server:
[svcscm@hg009.cln2 /data/scm/fbsource]$ hg config | grep automation
hooks.automationbookmarks=fbandroid/stable\nfbcode/stable\nfbcode/warm\nfbobjc/stable\nfbsource/stable
The usernames came from the disable-nonff.py hook on hg servers.
Reviewed By: suitingtseng, StanislavGlebik
Differential Revision: D16960784
fbshipit-source-id: 9cf63b40492431bf34fb0574738794b0190703c6
Summary:
If an error happens in CreateChangeset::create when we are committing a stack,
the parent ready channel immediately signals, and we end up with an error
that's very unhelpful: "Canceled". This happens because signal_parent_ready
gets dropped.
This patch updates the code to try and report what caused the parent to fail.
We can't clone errors, so all we show is a Debug representation of the error.
In practice, this causes the failure signal to be sent _after_ the parent data
future fails, so we actually see the parent error directly, which is just as
nice.
Reviewed By: farnz
Differential Revision: D16961035
fbshipit-source-id: 8cf3e78ecf785c471425e81f22a67a54d8bc3aeb
Summary:
The high level goal is to test new unodes functionality. In particular, we want
to test `list_directory()` functionality.
There's a problem though - `list_directory()` returns Nodehash and filetype for
each entry it returns. That's cheap to do with mercurial manifests but not
cheap with unodes:
1) Mercurial manifests store filetype (i.e. whether it's a file,
symlink or executable) in manifests. But we don't store it in unode manifests
(it's stored in a separate file unode entry).
2) Unodes don't have mercurial hashes at all.
However there's currently a single user of `list_directory()` call - this is
scmquery - and it just discards both of the fields - https://fburl.com/8vr5hibo.
So I decided to introduce a separate `list_directory_unodes()` method that
returns a list of directory entries without any additional info. The plan is to
use it in scmquery for repos that already has unodes backfilled. That's
intended to be short-term - ideally we replace current thrift interface with
something more modern or at the very least replace `list_directory` with
`list_directory_unodes()` completely.
Reviewed By: farnz
Differential Revision: D16918232
fbshipit-source-id: f05185be8eecbf0852c7ddb0a482d4a5a63f8b53
Summary:
This updates the `mononoke_admin` filenodes command to allow showing filenodes
by path + ID in addition to revision + path. This makes it possible to locate
the linknode given a filenode ID but no revision.
This updates the filenodes CLI command a little bit (so if you want the old
command you need to add `by-revision`), but this should be fine since this is
a command that a human will normally be calling.
Reviewed By: farnz
Differential Revision: D16936628
fbshipit-source-id: 1079ad580b235076739c92d0c1f3c9bfc80db4c1
Summary: This is needed for commit syncing between repos (in particular, stack beginning with D16896918).
Reviewed By: farnz
Differential Revision: D16919646
fbshipit-source-id: 37e09417bab033427479cfd8373e1e4a415a92f5
Summary: This adds a command that allows importing a set of LFS blobs into Mononoke by streaming them out of another storage location (e.g. Dewey).
Reviewed By: HarveyHunt
Differential Revision: D16917806
fbshipit-source-id: 4917d56e11a187c89e00c23a32c6e791b351f8ef
Summary:
Current pushrebase starts by mapping your `HgChangeset`s to `BonsaiChangeset`s. Cross-repo sync will generate bonsai form directly, then pushrebase it.
Reuse the existing pushrebase machinery - this also puts into a good place to start removing HgChangeset knowledge from pushrebase.
Reviewed By: StanislavGlebik
Differential Revision: D16915565
fbshipit-source-id: f6a1e77453eece047c1f3586ce659f9c88cc81e1
Summary: When we sync commits, we will need to be able to cross-reference the commits between the source repo and the target repo. Make it easy to track this mapping.
Reviewed By: StanislavGlebik
Differential Revision: D16896918
fbshipit-source-id: 3738cf2067291f16ce9b62509763ac526d1d35c2
Summary:
We needed this hack because in derive_unode_manifest crate we needed to implement Loadable trait (previously defined in
blobstore crate) for ManifestUnodeId which was defined in mononoke_types crate.
Rust doesn't let us do that - see
https://stackoverflow.com/questions/35160995/implement-foreign-trait-for-foreign-type.
So we used this new Id type, which made it awkward to use. This diff fixes it.
Reviewed By: krallin
Differential Revision: D16885044
fbshipit-source-id: 329b09bb3b671a59824d1d2597a3907bebb82642
