Summary:
Implement BlobstorePutOps for S3Blob. This uses is_present to check the various put behaviours
While implementing this I noticed get_sharded_key could be updated to take a reference, so I did that as well.
Differential Revision: D24079253
fbshipit-source-id: 16e194076dbdb4da8a7a9b779e0bd5fb60f550a6
Summary: Now that fileblob and memblob support put behaviour logic, update the overwrite test to check the overwrite result.
Differential Revision: D24021167
fbshipit-source-id: d9578630205cf5d79999a459cc29481968d5717d
Summary: Update memblob to be PutBehaviour aware by changing implementation from Blobstore to BlobstoreOps
Differential Revision: D24021166
fbshipit-source-id: 04dd25c5535769ea507120c1886592b808a7bbc6
Summary: Update Memblob::new callsites to ::default() in preparation for adding arguments to ::new() to specify the put behaviour desired
Differential Revision: D24021173
fbshipit-source-id: 07bf4e6c576ba85c9fa0374d5aac57a533132448
Summary: Add put behaviour handling to fileblob so that it can prevent overwrites if requested.
Differential Revision: D23933228
fbshipit-source-id: 8e74ac96b232be841174f6ad2bd2fccf92aaa90d
Summary:
Add put behaviour to BlobstoreOptions in preparation for passing in the put behaviour through blobstore_factory.
Later in the stack a command line option is added to set this non-None so that we can turn on overwrite logging for particular jobs.
Reviewed By: StanislavGlebik
Differential Revision: D24021169
fbshipit-source-id: 5692e2d3912ebde07b0d7bcce54b79df188a9f16
Summary: Add a new "user" column to the EdenAPI server's Scuba data.
Reviewed By: krallin
Differential Revision: D24153765
fbshipit-source-id: 95a3177d9283e5b0e3f47e7f42a1de5142049b99
Summary: Add a method to get the client's username from a client identity. This is helpful for logging, etc.
Reviewed By: krallin
Differential Revision: D24153766
fbshipit-source-id: 2ecf85e5de72918aeb292ce88539a991da4de900
Summary: Allow source control service clients to set pushvars.
Reviewed By: krallin
Differential Revision: D24136870
fbshipit-source-id: 34f9176ec66ca319b363c91015dae9b59a55a244
Summary:
Add the possibilty of setting `pushvars` when deleting bookmarks. This makes
it consistent with the other write operations.
Reviewed By: krallin
Differential Revision: D24136869
fbshipit-source-id: f98b74c6c731e50552184000ad697d04748711fd
Summary:
Previously all mutable_counters reads were going to leader. This might be
useful for some cases, but in the case of blobimport warmer this opens too
many connections to sql leader.
Let's read from replica instead
Reviewed By: krallin
Differential Revision: D24160315
fbshipit-source-id: 0cecde3c54a00bbea215a1e0fa63d4a7c3dc9eaa
Summary:
This seems to have broken as a result of a land race between D23999804 (6421dca639) and
D23455274 (bdff69b747). Let's fix it.
Reviewed By: ikostia
Differential Revision: D24158809
fbshipit-source-id: 1d733e2c93eb8a0803395d409fbb15e2e2146bdd
Summary: Adds version of `bounded_traversal_stream` where unfold returns a stream over children instead of an iterator. This function also applies back pressure on children iteration when we have too many unscheduled items.
Reviewed By: krallin
Differential Revision: D23931035
fbshipit-source-id: 2e2806653782d4e646dcdf4b2d4e624fd6543da8
Summary: Add `--debug` flag to `read_res cat` command for debug printing entire entry rather than just the data blob.
Reviewed By: kulshrax
Differential Revision: D23999804
fbshipit-source-id: 6955854edab2643cffbe5fae484a398716b48055
Summary:
Introduce `FileMetadata` and `DirectoryMetadata` to `Treeentry`, along with corresponding request API.
Move `metadata.flags` to `file_metadata.revisionstore_flags`, as it is never populated for trees. Do not use `metadata.size` on the wire, as it is never currently populated.
Leaving `DirectoryMetadata` commented out temporarily because serde round trips fail for unit struct. Re-introduced with fields in the next change in this stack.
Reviewed By: DurhamG
Differential Revision: D23455274
fbshipit-source-id: 57f440d5167f0b09eef2ea925484c84f739781e2
Summary:
This diff makes it so that pushrebase fails if tries to rebase over a commit
with a specified extra "failpushrebase" set. If a client runs into this issue
then they need to do a manual rebase.
Differential Revision: D24110709
fbshipit-source-id: 82cd771c92b9fb45f4fa8794b2c736f08ac900b1
Summary:
This is the first part of allowing us to update mononoke blobstore put behaviour to optionally a) log when it is overwriting keys, and b) not overwrite existing keys.
Introduce BlobstorePutOps for blobstore implementations so we can track overwrite status of a put, and force an explicit PutBehaviour if required. Its intended that only blobstore implementation code and special admin tooling will need to access BlobstorePutOps methods.
Reviewed By: farnz
Differential Revision: D24021168
fbshipit-source-id: 56ae34f9995a93cf1e47fbcfa2565f236c28ae12
Summary:
This passes `--tmpdir` option to `~/fbcode/eden/scm/tests/run-tests.py`
so it's predictable where for example mononoke's logs will be.
Some time ago I was debugging hanging test. It was very annoying that I couldn't specify that tmpdir manually. It also wasn't printed out (it's only printed out with `--keep-tmpdir` **after** the test finishes).
Now it is possible to specify that.
Reviewed By: krallin
Differential Revision: D24137737
fbshipit-source-id: 6280832517b48ece9b65e443c236035e385efea6
Summary:
This diff adds two things:
- the ability to compute the reverse of a `CommitSyncDataProvider::Test`, useful when creating both small-to-large and large-to-small `CommitSyncer` structs in tests
- the ability to set a current `CommitSyncConfigVersion` in the provider, which can also be useful, when simulating current version changes.
NB: I ended up not needing the set version functionality in my tests (further in the stack) in the end, so I can remove it, but I do think it will prove useful eventually.
Reviewed By: StanislavGlebik
Differential Revision: D24103206
fbshipit-source-id: 389169b2984684d83b0f6fdeb3be597d84cc0f12
Summary: Remove unnecessary clone in packblob along with the Clone constraint on the inner blobstore.
Reviewed By: krallin
Differential Revision: D24109293
fbshipit-source-id: b47e68e63b6ffda95d28d974ed6883e4ae31b3a1
Summary:
This is one more fix to use correct commit sync config version. In particular,
this diff fixes a case where a single parent commit was rewritten out. E.g.
if a large repo commit touches only files that do not remap in a small repo. In
that case we still want to record correct mapping so that all descendants used
the correct mapping as well.
Reviewed By: ikostia
Differential Revision: D24109221
fbshipit-source-id: bcdbb01b964d70227dff8363e77964716a345261
Summary:
Let's move initialization into a separate function. I'm planning to use it in
the next diff for another test
Reviewed By: ikostia
Differential Revision: D24109222
fbshipit-source-id: 73142dd46ef3de15ff381670ed6d5e31653c5dd4
Summary:
Previously fetch_bonsai_range returned all commits between `ancestor` and
`descendant`, but `ancestor` was included. This is usually not what we want and
it might be surprising and can lead to subtle bugs. As an example, next commit
in the stack might have failed pushrebases when it shouldn't do that.
This diff changes the semantic of the function to exclude an ancestor. This
function was used for 2 use cases:
1) Find changed files. find_rebased_set function was manually removing the
ancestor anyway, so there's no change in behaviour
2) To check that there are no case conflicts. Previously we were checking the
case conflicts with ancestor included, but that wasn't necessary. To prove that
let's go over the two possible situation:
i) This is a first iteration of the pushrebase
```
CB
SB |
| ...
... CA
SA
| /
root
```
in that case files introduced by root commit will be used to check if we have
case conflicts or not. But this is not necessary, because pushrebase assumption
is that CA::CB should not introduce any new case conflicts. Besides, even if
they added a case conflict then checking with just the files that were changed by root commit is
not enough to verify that.
Similar logic goes to SA::SB commits. Checking if root has any conflicts with
SA::SB commits doesn't make sense.
ii) This is not the first iteration of the pushrebase
```
CB
SB |
| ...
... CA
SA
|
O <- latest pushrebase attempt
... <- we rebased over these commits on the previous attempts
| /
root
```
In this case it's even easier. Commit O was verified on the previous iteration,
so no need to add it here again.
Reviewed By: aslpavel
Differential Revision: D24110710
fbshipit-source-id: 90dff253cba0013e9d5e401474132a152d473cae
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/65
Using $LOCALIP will ensure more consistent behavior when setting up the server in ipv4 or ipv6.
The LOCALIP variable was also abused when it was used to override ssh client address, so SSH_IP_OVEERIDE env was created here.
Lastly the result of `curl` call is now printed whenever the test failed to verify that Mononoke is running.
Reviewed By: farnz
Differential Revision: D24108186
fbshipit-source-id: e4b68dd2c5dd368851f0b00064088ffc442e31e0
Summary:
Normally, sync logic infers `CommitSyncConfigVersion` to use from parent commits (or from current version for root commits). However, for test purposes it is convenient to force a version override This logic does not change any of the production behaviors, and will be used in a later diff.
TODO: can it ever be needed beyond tests? I've thought about using this for "version boundary" commits, but those would probably just be constructed while completely bypassing the sync logic.
TBH, I am not certain this diff is a good change. But I've spend a very large amount of time crafting the repos used in the `sync_merge` tests later in this stack, so I am proposing to land this, then spend some time refactoring sync tests (and hopefully making it easier to craft test repos), then removing this logic. Obviously, this logic should only be landed if we land the tests in the first place.
Reviewed By: StanislavGlebik
Differential Revision: D24104101
fbshipit-source-id: 0825f04ed74532e89fd5f1fbebeee5f2001fedcd
Summary: It is sometimes very convenient to just inject new DAGs into existing repos.
Reviewed By: StanislavGlebik
Differential Revision: D24103164
fbshipit-source-id: abdfa18acb2f2fb1475b601a7eccb57e006982ec
Summary: No need to allocate a new vector if we just need to remove items from the current one.
Reviewed By: StanislavGlebik
Differential Revision: D24088319
fbshipit-source-id: 10804d925f20fe8dd1e2bb8500aa06d30bd367c1
Summary:
This just adds a single fn. I did not come up with a better place/name to put
it, suggestions are welcome. Seems generic enough to belong at the top-level
common location.
I've already needed this twice, so decided to extract. Second callsite will be further in the stack.
Reviewed By: StanislavGlebik
Differential Revision: D24080193
fbshipit-source-id: c3e0646f263562f3eed93f1fdbab9a076729f33c
Summary: `clippy` often complains about the use of `.len() != 0`, `.len() > 0` or `.len() == 0`and proposes to use `.is_empty()` instead. This diff does that across Mononoke.
Reviewed By: aslpavel
Differential Revision: D24099427
fbshipit-source-id: 1bba2f958485b7efb3f41bf3eae820879c92b0e5
Summary:
We want to end up with two `put` behaviours - overwrite and do not overwrite.
Currently, SQLBlob only implements the latter, but some users assume that `put` always overwrites. Change to match Manifold
Reviewed By: aslpavel
Differential Revision: D24079501
fbshipit-source-id: f75cac81acf874337c38f82597aae645c41a319b
Summary: Now that there are no more use-cases of `get_one`, let's remove it completely.
Reviewed By: farnz
Differential Revision: D24027990
fbshipit-source-id: 47baa6b1e28eedd94d95808efca0a98007a1d388
Summary:
This is a bit of a cargo-cult diff: it replaces the uses of `get_one` with `get` in tests, just to make the same wrong decisions later - use the first item from the produced list of items. So the only thing it does it removes a call site for `get_one`.
The reason it is ok to do `.into_iter().next()` here is because these are tests and we control the situation precisely - we know that there will be one mapping. Same reason we use `.unwrap()` in tests.
Reviewed By: farnz
Differential Revision: D24027785
fbshipit-source-id: 1c11acadfc9f7c6c4af658b414589c32008a6cce
Summary:
`get_one` is a deprecated method, because it uses incorrect logic to resolve ambiguities of multi-mapped commits: if just selects the very first of the potentially many mappings.
Correct resolution is to either handle the ambiguity at the caller site, or rely on provided resolution logic in commit_sync_outcome.rs.
Therefore, I am removing the uses of this method in this and a few surrounding commits.
In this case, the simplest thing is to replace it with `.get` and deal with multi-mappings on the client side:
- for `crossrepo map` subcommand we just print all mappings
- for `update_large_repo_bookmarks` we just fail on multi-mapping, as it seems dangerous to proceed without human intervention
Reviewed By: farnz
Differential Revision: D24030033
fbshipit-source-id: c84613579fbf8a5f6bac3c06da0cd4e0ad6c3fb0
Summary:
`get_one` is a deprecated method, because it uses incorrect logic to resolve ambiguities of multi-mapped commits: if just selects the very first of the potentially many mappings.
Correct resolution is to either handle the ambiguity at the caller site, or rely on provided resolution logic in `commit_sync_outcome.rs`.
Therefore, I am removing the uses of this method in this and a few surrounding commits.
In this case, I am changing `get_one` to `CommitSyncer::get_commit_sync_outcome`. There's no functional difference, as this is large-to-small mapping, which is always 1:1. But it allows us to get rid of `get_one` call-site, so let's do that.
Reviewed By: farnz
Differential Revision: D24027130
fbshipit-source-id: e57cb32c37a68e6762da6e2096ba216d251524f4
Summary:
`get_one` is a deprecated method, because it uses incorrect logic to resolve ambiguities of multi-mapped commits: if just selects the very first of the potentially many mappings.
Correct resolution is to either handle the ambiguity at the caller site, or rely on provided resolution logic in `commit_sync_outcome.rs`.
Therefore, I am removing the uses of this method in this and a few surrounding commits.
In this case, we can just rely on a provided `CommitSyncer::commit_sync_outcome_exists` method.
Reviewed By: farnz
Differential Revision: D24026470
fbshipit-source-id: 9f150eb3d6c39a58bb4b0d16d4cf18c324359013
Summary:
See D23991178 (87f2e4d0f8) for more details on the `CandidateSelectionHint`.
This diff adds integration test coverage for this functionality.
Reviewed By: farnz
Differential Revision: D24025165
fbshipit-source-id: 0ce70fe4c6b7347061a4815e49c0a1311e5964fa
Summary:
Interngraph token is already stored within keychain service. We should make use of that.
I'll need to remove related config option in convigerator in a separate diff.
Reviewed By: krallin
Differential Revision: D24015463
fbshipit-source-id: 9e8246e2cc252f0c42669140de7b50410a15709c
Summary:
This diff adds an ability to optionally pass a `CandidateSelectionHint` to `scs` implementation of the `xrepo-lookup` call, which would help in cases when ancestor commits have multiple mappings in the large repo.
Adding this functionality to `scsc xrepo-lookup` is essentially a way to manually fix multi-mapping problems, which could otherwise block Mononoke progress.
For more information on multi-mapping problems, see https://fburl.com/gmywf2d6.
TLDR is that `synced_commit_mapping` is `1:n` with `n` on the large repo side. When syncing commits, we need a way to disambiguate multi-mapped ancestors. `CandidateSelectionHint` is our way of doing this: it expressed desired properties of the commit we would like Mononoke to choose among the list of multi-mapped candidates.
Reviewed By: markbt
Differential Revision: D23991178
fbshipit-source-id: 29c90b7910ad1b84ff71964d6609521fded2f987
Summary: Building criterion in opt mode degradated enormously after moving to 0.3.3, pin it to 0.3.1 for now until we figure out what is the problem.
Reviewed By: ljw1004
Differential Revision: D24046885
fbshipit-source-id: 6373eb06b5f47061cc02597bf82f574511fbec43
Summary:
This will give us the host that hgcli is running on, which is, like, convenient
to know what hg host proxied a request. For comparison, currently, we have to
go through ssh prod logs using the client IP and port, and hope there aren't
too many matches, which is really not a reasonable way to debug things.
Reviewed By: ahornby
Differential Revision: D23994304
fbshipit-source-id: fa5b29aa50e278f0f1b3b60be42f634a1c5c45c1
Summary:
Remove assert_present from Blobstore trait as it had only one callsite other than the various blobstore layers/impls.
Replaced that one last call in repo_commit.rs/assert_in_blobstore() with an equivalent call to is_present.
Reviewed By: farnz
Differential Revision: D24016927
fbshipit-source-id: 764fddbebeb4b1192d196078b8824cf8a08e9691
Summary:
Let's make this configurable so we can control how many fallbacks we want to
allow if we're overloaded.
Reviewed By: farnz
Differential Revision: D24017088
fbshipit-source-id: 9bccaf831a28daff9696950ae8aac9d53e0c51c0
Summary:
The protocol for getpack is length-prefixed. However, we currently advertise
the number of characters in filenames instead of their byte length. So, the
lengths we send don't necessarily correspond to the amount of data we send.
Indeed, if a filename contains multibyte characters, we'll advertise a lower
byte count than what we actually end up sending. This results in the last
byte(s) of the filename being interpreted by Mononoke as the start of another
piece of data, and eventually causes Mononoke to hang as it waits for more data
that the client will never send.
This fixes that bug in reading, and also fixes an identical instance of the bug
on the server side. I also double checked the gettreepack code, which AFAICT
doesn't have this bug.
Reviewed By: ahornby
Differential Revision: D24013599
fbshipit-source-id: af716f2bf9c02d312c0c8d2f449988e8f8858ab8
Summary: Make the session banner a single line, and remove the URL.
Reviewed By: krallin
Differential Revision: D23930600
fbshipit-source-id: 1b361b9362f7652a2ad688ad599db2807d9344af
Summary: Make `parents`, `data`, and `metadata` optional, and introduce `WireTreeAttributesRequest` for selecting which attributes to request on the wire.
Reviewed By: kulshrax
Differential Revision: D23406763
fbshipit-source-id: 5edd674d9ba5d37c23b12ab4d7b54bbf6c9ff990
Summary:
Adds a `WireTreeQuery` enum for query method, with a single `ByKeys(WireTreeKeyQuery)` available currently, to request a specific set of keys.
Leave the API struct alone for now.
Reviewed By: kulshrax
Differential Revision: D23402366
fbshipit-source-id: 19cd8066afd9f14c7e5f718f7583d1e2b9ffac02
Summary:
This diff introduces Mysql client for Rust to Mononoke as a one more backend in the same row with raw xdb connections and myrouter. So now Mononoke can use new Mysql client connections instead of Myrouter.
To run Mononoke with the new backend, pass `--use-mysql-client` options (conflicts with `--myrouter-port`).
I also added a new target for integration tests, which runs mysql tests using mysql client.
Now to run mysql tests using raw xdb connections, you can use `mononoke/tests/integration:integration-mysql-raw-xdb` and using mysql client `mononoke/tests/integration:integration-mysql`
Reviewed By: ahornby
Differential Revision: D23213228
fbshipit-source-id: c124ccb15747edb17ed94cdad2c6f7703d3bf1a2
Summary: We use hyphens in other bypass strings. Make this consistent in `test-hooks.t` to avoid confusion.
Reviewed By: mitrandir77
Differential Revision: D23964799
fbshipit-source-id: e300bad091aa6c50f5921507117c1019b9863bd5
Summary:
Let's start actually fixing what commit sync config version is used to remap a commit i.e. we
should use a commit sync config version that was used to remap a parent instead
of using a current version. See more details in
https://fb.quip.com/VYqAArwP0nr1
This diff fixes one particular case and also leaves a few TODOs that we need to
do later
Reviewed By: krallin
Differential Revision: D23953213
fbshipit-source-id: 021da04b0f431767fec5d1c4408287870cb83de1
Summary:
TestLiveCommitSyncConfig is supposed to be a test replacement of
CfgrLiveCommitSyncConfig, however it was quite a different semantic. In
particular, it wasn't even possible to have two versions of the mapping for the
single repo.
This diff changes that. Now we'll have a method to add commit sync config
version, and mark/remove a version as current
Reviewed By: krallin
Differential Revision: D23951202
fbshipit-source-id: 242b4f088f67dac504544987e484cc290ee4e400
Summary: Instead of always fetching the current version name to verify working copy let's instead fetch whatever the version was actually used to create this commit.
Reviewed By: krallin
Differential Revision: D23936503
fbshipit-source-id: 811e427eb62741401b866970b4a0de0c1753edb3
Summary:
Turned out validation didn't report an error if source repo contained an entry
that was supposed to be present in target repo but was actually missing.
This diff fixes it.
Reviewed By: krallin
Differential Revision: D23949909
fbshipit-source-id: 17813b4ad924470c2e8dcd9d3dc0852c79473c61
Summary: Since now we store it in the db, let's also expose it in CommitSyncOutcome enum
Reviewed By: krallin
Differential Revision: D23936502
fbshipit-source-id: a0758143ceaa8f5706f1d9cfe3040ac91c7bac49
Summary: The diff adds API to create a set of connections: read, read master and write.
Reviewed By: ahornby
Differential Revision: D23568561
fbshipit-source-id: b3ee954604557497ed56c6b369256b6f76a1e042
Summary:
This diff makes blobstore healer to use MyAdmin to get replication lag for a DB shard and removes "laggable" interface for connections.
The old "laggable" API worked this way: we maintained potential connections to each possible region, then tried to query replica status on all of them. If there was no replica hosts in some of the regions, we just wanted to ignore it by handling a specific error type.
This is legacy and makes the logic more complicated. We want for the new code to use Myadmin instead.
Reviewed By: krallin
Differential Revision: D23767442
fbshipit-source-id: 9f85f07bd318ad020d203d2bcd1c8898061f7572
Summary:
I spotted D23955629 recently, and it turns out I made the same mistake in LFS.
Let's fix it. In order to keep the same semantics (yield on every chunk from
the filestore), I lowered the threshold a bit from what it was intended to be.
Reviewed By: johansglock
Differential Revision: D23960152
fbshipit-source-id: 4d384752228fd125ade7e032a477648798e7fa44
Summary:
This is (I think) the last step required to make rust-partial-io be github-first.
The diff was created using:
* `zbgs partial-io` and remove all instances of it in fbsource
* `hg rm common/rust/partial-io`
* add `partial-io` to `third-party/rust/Cargo.toml`
* `common/rust/tools/reindeer/vendor` to vendor
* `buck build eden/mononoke/mercurial/...` to check that it builds correctly
* `buck run //common/rust/cargo_from_buck:cargo_from_buck` to run autocargo
Reviewed By: aslpavel
Differential Revision: D23849634
fbshipit-source-id: 339fc3976cc9a0b6f10a0538d643b87797e2bc3c
Summary:
See motivation for the change in D23845720 (5de500bb99).
We'll need to store version name even for commits that weren't rewritten, but that have an equivalent working copy in another repo.
Reviewed By: ikostia
Differential Revision: D23864571
fbshipit-source-id: 408b68c3b0aa9885a9cd248b0b4abc2b87cd4cca
Summary:
get_source_target_mover likely awaits the same fate as
get_current_mover_DEPRECATED functions i.e. get_source_target_mover will likely
be removed.
This diff just removes a few intances of this function.
Reviewed By: ikostia
Differential Revision: D23929748
fbshipit-source-id: 2ac09da164de3916a552757acf0c39387f6126e4
Summary:
get_mover() and get_reverse_mover() functions return the mover for the
"current" version of the commit sync config, which means these are movers for the version
of the config that's used to create the latest commits on master branch.
So this function returns correct mover only for the latest master commit, but
for all other commits it returns an incorrect mover! This is wrong and it
happened to work just by change, and that's why these functions are marked as deprecated
now, and later we'll add functions 'get_mover_by_version()' which could be used to
replace deprecated functions.
Note that the story for get_bookmark_renamer()/get_reverse_bookmark_renamer()
functions seems to be different. If we can always figure out what's the correct
mover for a commit by e.g. look at its parent we can't really do the same for
bookmarks. Because of that I suggest to keep using the current version for
get_bookmark_renamer() function.
Reviewed By: ikostia
Differential Revision: D23929582
fbshipit-source-id: 3e5e9b46224aca0b75cf2d981ea21c4f9a378ba9
Summary: Finally remove version_name from CommitSyncRepos. Note that this diff adds a few TODOs that we'd need to fix later.
Reviewed By: ikostia
Differential Revision: D23929010
fbshipit-source-id: c72130af548ac7b26bc20ddaac9a59562cc75e0b
Summary: Just as in the previous diff, but this time remove bookmark_renamers from CommitSyncRepos
Reviewed By: ikostia
Differential Revision: D23910295
fbshipit-source-id: 0c2d147057c8d3e0749d5b31ef98ab5022255d95
Summary: Just as the previous diff, but this time it removes reverse_mover
Reviewed By: ikostia
Differential Revision: D23879509
fbshipit-source-id: ed111ca2d106120229c4facc0bb2435913c27966
Summary:
This diff starts to use CommitSyncDataProvider introduced in the previous diff
and removes Mover from CommitSyncRepos struct.
Reviewed By: ikostia
Differential Revision: D23878683
fbshipit-source-id: 0d54f889781aebe4726b3388343a87df783c17d4
Summary:
As described in D23845720 (5de500bb99) we are doing a pretty significant change in the
CommitSyncer. Previously it stored static Movers and BookmarkRenamers, but it
needs to change and they would need to fetch config from LiveCommitSyncConfig.
Unfortunately we already have a bunch of tests that create weird movers that
would be very hard to model via LiveCommitSyncConfig. So we have 2 options:
1) delete or rewrite all these tests
2) Create a wrapper that would return a mover/bookmark renamers.
Option #2 is preferable, and that's what this diff does. In production it would
use LiveCommitSyncConfig, but it also let's tests specify whatever weird movers
they need.
Reviewed By: ikostia
Differential Revision: D23909432
fbshipit-source-id: 83fb627812f625e07f7e40044e2f69274cd2d768
Summary:
Our configerator configs store both "current" mapping version and also they
store all versions that were used before.
These integration tests were updating just current version, but weren't
updating "all" versions. This is incorrect but it worked by accident.
But it will stop working in the next diffs, so this diff fixes it
Reviewed By: ikostia
Differential Revision: D23908970
fbshipit-source-id: 10f96bd02987d9195aff4855241efbd9a065a761
Summary:
A higher-level goal is to provide an interface for the manual remediation of
various xrepo-sync blockages. The nature of a candidate selection is such that
it may fail if the hint is not sufficient to decide which remapping changeset
is the best candidate. This is especially true about the `Only` hint variant:
it is designed to fail when there's more than one candidate. But even with
bookmark ancestorship hints, there are corner cases when the algorithm cannot
make a decision (as well as there may just be bugs in the algorithm). These
cases should be **extremely** rare. Nevertheless, we want to be able to unblock
ourselves. To do so, it is proposed to acccept parent selection hints in the `xrepo-lookup` scs
method. By default, it will use `Only` as a hint and be semantically equivalent
to the current behavior. But we'll provide CLI options to select other hints.
In order to make this work, we need the `sync_commit` method of the
`CommitSyncer` to accept hints too.
Reviewed By: StanislavGlebik
Differential Revision: D23913216
fbshipit-source-id: 05e1ff99cd2c6522829a6e8569040b226600af60
Summary:
This diff adds the use of candidate selection hints to `cross_repo_sync` code sights, which need to query `CommitSyncOutcome` in the small-to-large direction. Specifically: `unsafe_sync_commit` and `unsafe_sync_commit_pushrebase` are the two main functions.
One will now get `CandidateSelectionHint` from the callsight (most notably: `push_redirector`), the other one will build a bookmark-based hint itself.
Reviewed By: StanislavGlebik
Differential Revision: D23715259
fbshipit-source-id: 3f4924f1337b09f3762cc050c4017c5d2bd6cab6
Summary:
The remaining test failures are mostly around bundle support, which
I'll fix in a later diff.
Reviewed By: quark-zju
Differential Revision: D23664037
fbshipit-source-id: 2bdde3cb4fcded6e0cf3afdc23269662544821df
Summary:
ConnectionSecurityChecker now supports per repository ACL checking.
PermissionCheckers are created in constructor for each repo.
Later when there is a need to check permissions, they're retrieved using a hash map.
Reviewed By: HarveyHunt
Differential Revision: D23678515
fbshipit-source-id: 3d2880fc9df137872ea64a47636f1142d0b36fc1
Summary:
Previously we had just an empty TestLiveCommitSyncConfig in tests. Since we are
not using it at all right now, it was fine, but we are planning to start using
it later. To do that let's configure TestLiveCommitSyncConfig so that it's not
empty but actually stores a real content.
Reviewed By: ikostia
Differential Revision: D23903579
fbshipit-source-id: af05a377f730c1824b03327749e6f824361e23e2
Summary:
At the moment we have a weird setup where cross repo sync configuration is
stored in both live commit sync configuration and in normal mononoke config.
The latter is deprecated, however there are still a few parts of the codebase
that rely on that. This diff fixes one place
Reviewed By: ikostia
Differential Revision: D23903578
fbshipit-source-id: 2bf4b3d17c34fe2eb6330cd862f7b0f5cd6ffa40
Summary:
In D23845720 (5de500bb99) I described what changes we need to make in our commit syncer. One
part of it is that we should remove get_mover() method, as this method always
uses current version of commit sync map even, and that's incorrect.
This diff removes it from commit validator
Reviewed By: ikostia
Differential Revision: D23864350
fbshipit-source-id: 3f650a32835dda9f82949002d63b52cc36cf04e0
Summary:
D23599866 (54d43b7f95) added an optimization for getbundle that reduces cpu usage when a new
commit with log generation number is added. I.e. the case like this
```
O
|
O
..
O <- new commit, low generation number
|
...
```
Unfortunately this optimization doesn't help with the case where a new repo is
merged into master
```
O <- also new commit, but generationo number is high!
| \
.. O <- new commit, low generation number, but it's not in "heads" parameter
|
|
O
...
```
The merge commit actually has a high generation number, but it's p2 has a low
generation number, so it causes the same issue with high cpu usage.
This diff adds a second optimization ( :( ) that should help with the shortcoming of the first one. See comments for more details.
Reviewed By: ikostia
Differential Revision: D23824204
fbshipit-source-id: 8f647f1813d2662e41325829d05def633372c140
Summary:
This takes johansglock's D23757705 one step further, and gets rid of the
`Wait<...>` wrapper we use to synchronously write to stderr in our logging on
the Mononoke Server side.
This should be fine because We send very little logs to the client, so just
buffering them seems like it won't really hurt, and even if we were writing a
log, it certainly would hurt less than blocking our runtime threads into an
interruptible wait.
A problem is that we actually use this in hgcli, where we want to read from our
stdin and write to our stdout / stderr. Rather than port all this stuff, this
diff updates hgcli to just use Tokio's abstractions for stdink, stdout, and
stderr. I ported the various buffer sizes we use to use there in here (I think
we should buffer less from the server though — 50000 buffers is a lot).
I did however update this to write to `std::io::stderr()` instead of an async
stream for this. I think it's fine considering:
- Internally, Tokio also uses `std:io::stderr()` which has a lock on writing.
- We hardly write anything anyway
Reviewed By: StanislavGlebik
Differential Revision: D23762062
fbshipit-source-id: c8d5330b0735d47b6de00e1a54aee4fed97db6b0
Summary:
There were a few instances of timed out getpack requests on ovrsource.
Example: https://fburl.com/sandcastle/yylr1w3v
Let's bump the timeout to unblock them.
Reviewed By: krallin
Differential Revision: D23900374
fbshipit-source-id: 3ee6e2d4f6b6ed12cd0c1516c686a03c87fa7cb4
Summary:
CommitSyncer is a struct that we use to remap a commit from one repo to another. It uses commit sync map to figure out which paths need to be changed. Commit sync mapping might change, and each commit sync mapping has a version associated with it.
At the moment CommitSyncer doesn't work correctly if a commit sync mapping is changed. Consider the following DAG
```
large repo
A' <- remapped with mapping V1
|
O B' <- remapped with mapping V1
| /
...
small repo
A
|
O B
| /
...
```
We have commit A and B from a small repo remapped into a large repo into commits A' and B'. They were remapped with commit sync mapping V1, which for example remaps files in "dir/" into "smallrepo/dir".
Now let's say we start to use a new mapping v2 which remaps "dir/" into "otherdir/". After this point every commit will be created with new mapping. But this is incorrect - if we create a commit on top of B in a small repo that touches "dir/file.txt" then it will be remapped into "otherdir/file.txt" in the large repo, even though every other file is still in "smallrepo/dir"!
The fix for this issue is to always use the same mapping as commit parent was using (there are a few tricky cases with merge commits and commits with no parents, but those will be dealt with separately).
This diff is the first step - it threads through LiveCommitSyncConfig all the way to the CommitSyncer object, so that CommitSyncer can always fetch whatever mapping it needs.
Reviewed By: ikostia
Differential Revision: D23845720
fbshipit-source-id: 555cc31fd4ce09f0a6fa2869bfcee2c7cdfbcc61
Summary:
Our current megarepo configuration is in a bit of a mess:
1) We have LiveCommitSyncConfig, which fetches the latest version of configs
from configerator and should be used in all cases
2) However we still have an old commit that's stored in mononoke config. It
shouldn't really be used at all.
Unfortunately there are a few places where #2 is still used. This diff removes
one of them.
Reviewed By: ikostia
Differential Revision: D23845297
fbshipit-source-id: aa2d591223cc4b8fe5ef264147457fcb3d1faad7
Summary:
Introduce separate wire types to allow protocol evolution and client API changes to happen independently.
* Duplicate `*Request`, `*Entry`, `Key`, `Parents`, `RepoPathBuf`, `HgId`, and `revisionstore_types::Metadata` types into the `wire` module. The versions in the `wire` module are required to have proper `serde` annotations, `Serialize` / `Deserialize` implementations, etc. These have been removed from the original structs.
* Introduce infallible conversions from "API types" to "wire types" with the `ToWire` trait and fallible conversions from "wire types" to "API types" with the `ToApi`. API -> wire conversions should never fail in a binary that builds succesfully, but wire -> API conversions can fail in the case that the server and client are using different versions of the library. This will cause, for instance, a newly-introduced enum variant used by the client to be deserialized into the catch-all `Unknown` variant on the server, which won't generally have a corresponding representation in the API type.
* Cleanup: remove `*Response` types, which are no longer used anywhere.
* Introduce a `map` method on `Fetch` struct which allows a fallible conversion function to be used to convert a `Fetch<T>` to a `Fetch<U>`. This function is used in the edenapi client implementation to convert from wire types to API types.
* Modify `edenapi_server` to convert from API types to wire types.
* Modify `edenapi_cli` to convert back to wire types before serializing responses to disk.
* Modify `make_req` to use `ToWire` for converting API structs from the `json` module to wire structs.
* Modify `read_res` to use `ToApi` to convert deserialized wire types to API types with the necessary methods for investigating the contents (`.data()`, primarily). It will print an error message to stderr if it encounters a wire type which cannot be converted into the corresponding API type.
* Add some documentation about protocol conventions to the root of the `wire` module.
Reviewed By: kulshrax
Differential Revision: D23224705
fbshipit-source-id: 88f8addc403f3a8da3cde2aeee765899a826446d
Summary:
At the moment CommitSyncConfig can be set in two ways:
1) Set in the normal mononoke config. That means that config can be updated
only after a service is restarted. This is an outdated way, and won't be used
in prod.
2) Set in separate config which can be updated on demand. This is what we are
planning to use in production.
create_commit_syncer_from_matches was used to build a CommitSyncer object via
normal mononoke config (i.e. outdated option #1). According to the comments it
was done so that we can read configs from the disk instead of configerator, but
this doesn't make sense because you already can read configerator configs
from disk. So create_commit_syncer_from_matches doesn't look particularly
useful, and besides it also make further refactorings harder. Let's remove it.
Reviewed By: ikostia
Differential Revision: D23811090
fbshipit-source-id: 114d88d9d9207c831d98dfa1cbb9e8ede5adeb1d
Summary:
There's a [flaw](https://fb.workplace.com/groups/scm.mononoke/permalink/1220069065022333) in the current `synced_commit_mapping` data model. In a nutshell, the flaw is in the assumption that the `RewrtittenAs` relationship is `1:1`, while in fact it is `1:n` with `n` on the large repo side.
To address this flaw I propose to:
- relax the DB constraints to represent the semantically correct model
- select all the synced candidates from the DB
- for places in code, which require a single mapping for a candidate, use the provided hint to resolve any ambiguity
More concretely:
- instead of a single `CommitSyncOutcome` struct, I propose to have the "canonical" `PluralCommitSyncOutcome` and the "resolved" `CommitSyncOutcome`
- every variant of `PluralCommitSyncOutcome` that is not `RewrittenAs` just maps to an identical variant of `CommitSyncOutcome`
- have a `CandidateSelectionHint` passed from the clients, which would help resolve `PluralCommitSyncOutcome::RewrittenAs` into a `CommitSyncOutcome::RewrittenAs`
- if the hint does not help to resolve `PluralCommitSyncOutcome::RewrittenAs` into an unambiguous `CommitSyncOutcome::RewrittenAs`, just fail the request and require human intervention to deal with things
- within the hint, have for the following variants for the resolution algorithm:
- `Only` which fails the resolution if there's more than one candidate
- `Exact` behaves like `Only` if there's one candidate, otherwise selects a provided candidate
- `OnlyOr(Ancestor|Descendant)Of(Commit|Bookmark)` behave like `Only` if there's one candidate, otherwise select a candidate in the expected topological relationship
Note some important decisions, that may be surprising at first:
- if there's just one candidate, resolutions with all types of hints succeed, even if this candidate does not fit the hint otherwise (for example, if the hint is `Exact(A)`, and the list of candidates is `[B]`, the resolution succeeds.
- for bookmark-related hints, if the bookmark does not exist at the time of resolution, the hint just "downgrades" itself to be `Only`
Both of these emphasize the fact that if the mapping has only one `RewrittenAs` candidate for a given changeset, the behavior does not change.
Reviewed By: StanislavGlebik
Differential Revision: D23670180
fbshipit-source-id: 1cee1f65fc8020e0ae8a7da789b2532d2e436b77
Summary: SomeFailedOthersNone should not consider write mostly blobstores None if all other stores Error
Reviewed By: farnz
Differential Revision: D23840334
fbshipit-source-id: 9838bead6fec0d5f920e4a788387025d0dacf80b
Summary: Add a test for SomeFailedOthersNone when write mostly blobstore is None
Reviewed By: farnz
Differential Revision: D23840685
fbshipit-source-id: 81834663169b3a522b9c08e0a36f0b91354916c7
Summary:
Let's add a command that validates that the created catchup commit is correct.
For now it validates that unodes are the same between catchup commit and commit
that we are merging in.
Later we can add more invariants that we want to check.
Reviewed By: krallin
Differential Revision: D23782369
fbshipit-source-id: 61d19aa73777d5fbb3e1b127bdcf39f5e6309b52
Summary: Add error context to file content scrub so that we can tell if an Error has propagated via the scrub stream loading.
Reviewed By: StanislavGlebik
Differential Revision: D23838144
fbshipit-source-id: 40a8a090510959cab1020182c19076b8a3317b1b
Summary:
Implemented S3 blobstore
Isilon implements S3 as 1:1 mapping into filesystem, and it limits the maximum number of blobs in the single directory. To overcome it lets shard the keys using base64 encoding and making 2 level dir structure with 2 chars dir names.
Reviewed By: krallin
Differential Revision: D23562541
fbshipit-source-id: c87aca2410381a07babb191cbd8cf28233556e03
Summary:
When running the repo import tool, it's possible that we need to do additional setup steps before being able to run the tool, which otherwise would only come up when we run it.
Firstly, if the repo we import into doesn't have a callsign (e.g. FBS, WWW...), but we want to check Phabricator, our tool would hang when checking Phabricator, because we need the callsign for checking. Therefore, we need to inform the user to set the callsign for the repo.
Secondly, in case the repo push-redirects to a larger repo, we generate a bookmark for the commits imported into the large. However, we need to inform the Phabricator team to include the large repo's bookmark before we can import the commits, because this bookmark publishes the imported commits on Phabricator.
This diff adds a subcommand to check these additional steps, so we wouldn't find these out during the actual import run.
Reviewed By: StanislavGlebik
Differential Revision: D23783462
fbshipit-source-id: 3cdf4035548213d8cee9717fb985c22741a6749b
Summary:
In the later diffs we are going to change how CommitSyncer is initialized. In
order to make it simpler let's refactor cross_repo_sync_test to move
CommitSyncer creation in a single function.
There are a few tests that have very peculiar initialization - for example they
have movers that fail. For those tests I combined the new function for creation
of CommitSyncer with manual initialization of CommitSyncRepos struct.
Reviewed By: krallin
Differential Revision: D23811507
fbshipit-source-id: 682ab30aa09c9189fcd02850a19f1ddf021c0329
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/60
For the tests that output different data to stdout in OSS vs FB create helpers that remove the differences.
Reviewed By: farnz
Differential Revision: D23814134
fbshipit-source-id: c6656528021c9a90b98e3c89a9bbe8c5178c6919
Summary:
Add `scsc land-stack` to facilitate testing of stack landing via the source control service.
Use this to test that landing of stacks works.
Reviewed By: aslpavel
Differential Revision: D23813366
fbshipit-source-id: 1f7b682fa5e33a232cb1da5c702a703223658942
Summary:
Update the conversion of `BookmarkMovementError` to `MononokeError` to reflect
that most movement errors are caused by invalid requests.
Reviewed By: aslpavel
Differential Revision: D23814794
fbshipit-source-id: 48503353aaae7b3cd03e5221a8ad014eef2e9414
Summary:
Implement the `repo_land_stack` method by working out which commits are in the
stack to be landed, and then pushrebasing them onto the target bookmark.
Reviewed By: aslpavel
Differential Revision: D23813370
fbshipit-source-id: babe34f0e9f1db055adb2e5d1debefd8ebcf6f86
Summary:
Sometimes the `AsyncIntoResponse` trait needs additional data (e.g. the set of commit
identity schemes the client is interested in) to convert the item into the response
type.
Currently we use a tuple of `(item, &additional_data)` to work around this, however
this will become less readable as we add new items with more additional data.
Split this use-case out into a new trait: `AsyncIntoResponseWith`. This defines
an associated type which is the type of the additional data needed, and provides a
new method `into_response_with`, where a reference to the additional data can be
provided.
Note that conversions for tuple types that are logical `(name, value)` or `(id,
value)` pairs are still ok. It is specifically the case where we have `(item,
&additional_data)` that we are converting here (i.e. the additional data merely
informs the conversion, it is not part of the resulting response value).
Reviewed By: aslpavel
Differential Revision: D23813371
fbshipit-source-id: c0dcfe826288ad53ad572ae4dd956540605998f5
Summary: Make it clear which error is which, and what the number of expected and actual items are.
Reviewed By: StanislavGlebik
Differential Revision: D23813369
fbshipit-source-id: 5b94c5a67438c475235876669ec2be3fd1866700
Summary: Intern ids to reduce space used in the walk state. This is significant on large repos.
Reviewed By: farnz
Differential Revision: D23691524
fbshipit-source-id: b42f926d88083d06ffc44508db44747f9a14e0a5
Summary:
Passing option is not necessary since live_commit_sync_config is always
available.
Reviewed By: ahornby
Differential Revision: D23811021
fbshipit-source-id: ee11f88d57814d9abac8650e52febd9e431770da
Summary:
I've re-backfilled some of blame values for configerator. But old values might
still be in memcache. To make sure that's not the case let's bump the memcache
key.
Reviewed By: krallin
Differential Revision: D23810971
fbshipit-source-id: c333a51ffb2babf7da808b276f9cfa31baaa105c
Summary: Small change to make it more readable and reduce likelihood of allocation (although the collect might be optimized away anyway)
Reviewed By: farnz
Differential Revision: D23760762
fbshipit-source-id: 5c47352386de128b65052d63b3f3ff1081a462e3
Summary: Make `StreamBody` accept a `Stream` of `Bytes` instead of a `TryStream` of `Bytes`. This means that applications returning streaming responses will be forced to deal with errors prior to returning the response.
Reviewed By: krallin
Differential Revision: D23780216
fbshipit-source-id: dbad61947ef23bbfc4edf3d286ad0218c1859d87
Summary:
Using the `EndOnErr` combinator introduced in the previous diff, log any errors that occur during a streaming response to stderr.
Note that **the intent of this diff is to implement the most basic possible example of doing something with these errors**, with the goal of allowing us to modify `StreamBody` to only accept infallible `Stream`s.
I'd imagine that in all likelihood we'd want to do something smarter with the errors than just print them, but I figure that can be added in later diffs since it seems like doing something else (like logging the error to Scuba or adding to the RequestContext) might require additional changes that are beyond the scope of this diff.
At the very least, this seems like an improvement from before, where these errors would just get passed straight through to Hyper.
Reviewed By: krallin
Differential Revision: D23780217
fbshipit-source-id: 2f885f9fdc6af3dd28d95be1daa1d82c732453fa
Summary: Add a new `EndOnErr` combinator for `TryStream`s (exposed via the `GothamTryStreamExt::end_on_err` method) which fuses the underlying `TryStream` upon hitting an error, and passes the error to the user-provided callback. This is useful for contexts like the LFS server, where mid-stream errors are considered unrecoverable and must result in termination of the response.
Reviewed By: krallin
Differential Revision: D23778490
fbshipit-source-id: 05caa52ca62d085cc63cc8feb4619188fe0fac61
Summary: Use the new `ForwardErr` stream combinator to log errors that occur during a streaming response. Right now, they are just printed to stderr, but in the future we should also do other things such as logging them to Scuba. This diff supersedes the approach from D22720957.
Reviewed By: krallin
Differential Revision: D23780215
fbshipit-source-id: 8d2267f1166e665a62a167a6d95bb0b1797b5767
Summary: Implement `ContentMeta` for streams wrapped with the `ForwardErr` combinator, so that they may be used as input to `StreamBody`. (Presently, this won't actually work since `StreamBody` expects a `TryStream`, but this will change later in the stack.)
Reviewed By: krallin
Differential Revision: D23777842
fbshipit-source-id: 234bcdf104afbf2c9832fbe54d85744bfb470363
Summary:
This diff adds a new `ForwardErr` combinator that allows redirecting the errors from a `TryStream` into a channel, allowing them to be processed asynchronously without disrupting the stream itself. This effectively splits the `TryStream` into two `Stream`s containing the successful items and errors respectively.
To make it easy to use the combinator, I've added a new `GothamTryStreamExt` extension trait (in the vein of the old `futures_ext` crate) that allows the user to simply call `forward_err` on any `TryStream`. The trait name is a bit of a misnomer (suggestions welcome!) in that there isn't anything Gotham-specific about it, but the name `TryStreamExt` is taken and I didn't want to set up a successor to `futures_ext` just for the sake of one combinator. (Though we will likely expand the trait in the future.)
Reviewed By: krallin
Differential Revision: D23777501
fbshipit-source-id: 8628cdc2e104cd9b972afda745858f9cb9e85245
Summary:
Compressed responses from LFS are slower than they should right now. Normally,
we'd expect something along the lines of normal response time + compression
time, but right now it's a lot more than this.
The reason for this is that our compressed streams are eager, i.e. they will
consume and compress as much of the underlying stream as possible before
sending off the data. This is problematic for LFS, because we try very hard to
serve everything out of RAM directly (and very often succeed), so that means
we compress the whole stream before sending it off.
This means we might spend e.g. 500ms compressing (this is how long it takes
zstd to compress the object I was testing on, which is a ~80MiB binary that
compresses down to 33% of that), and _then_ we'll spend some time transferring
the compressed data, when we could have started transferring immediately while
we were compressing.
To achieve this, let's simply tell our compressed stream to stop waiting for
more data once in a while (every 4 MiB, which seems very frequent but actually
really isn't).
Reviewed By: StanislavGlebik
Differential Revision: D23782756
fbshipit-source-id: a0d523d84f92e215eb366f551063383fc835fdd6
Summary:
I saw this throw the LFS server into an infinite loop when I tested it. We're
not using this right now, so I'm not investing time into root-causing the
issue, and instead let's just take this out.
Reviewed By: StanislavGlebik
Differential Revision: D23782757
fbshipit-source-id: f320fc72c3ff279042c2fe9fcb9c4904e9e1bfdf
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/51
This diff extends capabilities of CargoBuilder in getdeps so that individual manifests can be build even without workspaces. Thanks to that a build for edenapi/tools can be made and its artifacts can be used in mononoke integration tests.
Reviewed By: StanislavGlebik
Differential Revision: D23574887
fbshipit-source-id: 8a974a6b5235d36a44fe082aad55cd380d84dd09
Summary:
This makes deletion commits a bit less confusing, but it also have another
benefit.
Without the sort some directories might have been changed multiple times in
deletion commits e.g. if a directory had 5 files, and these files were deleted
in 5 different deletion commits then the directory would be changed 5 times.
This was not good, because it made some data derivation slower (in particular,
fastlog), because it had to regenerate the same data over and over again.
Reviewed By: ikostia
Differential Revision: D23780066
fbshipit-source-id: d5c52b13f58dcaf2012d9c12bf77398561cf10ef
Summary:
Spotted a TODO in fsnode get_fsnode_id. There was only one user of the function, which didn't really need to call it as it had the blob already.
As well as being a bit tidier this also saves a clone of the fsnode.
Reviewed By: StanislavGlebik
Differential Revision: D23758689
fbshipit-source-id: e0a8c124c929fda3af4c96a76d441a79e5bfbd5b
Summary:
Save memory in walker state tracking by not memoing hash values. For large repos this is significant.
I was expecting a small slowdown from this, but so far looks pretty much the same walk rate. Speculation: this may be due to the num cpus lock sharding fix in dashmap 3.11.10 which means there are many more shards than when the memo was tested with 3.11.9, so saving time inside locks is less significant.
Reviewed By: StanislavGlebik
Differential Revision: D23680550
fbshipit-source-id: 351b5ec39885fc30996207c7dccc22c749e30321
Summary:
The `gotham_ext::response` module was getting a bit large, so this diff moves `ContentMeta`, `ContentStream`, and `CompressedContentStream` into a new submodule, alongside the contents of the old `content_encoding` module. This way, the `response` module remains entirely centered around the `TryIntoResponse` trait (and the various body structs that implement that trait).
Later diffs in this stack will be adding an additional layer between the content streams and the body structs, at which point it probably doesn't make sense to have these right next to each other. Splitting them out now will allow for better code organization going forward.
Reviewed By: krallin
Differential Revision: D23777492
fbshipit-source-id: 86e598dcb37578d3b22217a2a65f1bde84d72215
Summary: Previously, we used the call sign of the repo we import when checking any of the if the commits are parsed by Phabricator. However, we also use this callsign for other repos when checking Phabricator, which is an incorrect implementation. E.g. if fbsource back-syncs to ovsource, we would have used FBS callsign when checking Phabricator for both fbsource and ovrsource, but we should use OVRSOURCE callsign for repo ovrsource. This diff corrects this implementation by saving the callsigns of the small repos in their SmallRepoBackSyncVars.
Reviewed By: StanislavGlebik
Differential Revision: D23758355
fbshipit-source-id: b322acb2ec589eabed5362bfd6b963e2dd1d6ea9
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/58
This makes the test-bookmarks-filler.t pass. Additionally remove few tests from exclusion lists as they started to pass.
Reviewed By: ikostia
Differential Revision: D23757401
fbshipit-source-id: eddcda5fd1806d77d0046b6ced3695df6b3d775d
Summary:
We are running out of space on integration tests runs on Linux. In order to avoid that this change is adding some cleanups.
1. Adding `docker rmi $(docker image ls -aq)` frees up 4 GB.
2. Cleaning up `eden_scm` build directory frees up 3 GB.
3. Cleaning up `mononoke` build directory frees up 1 GB.
This diff also includes a fix for run_tests_getdeps.py where we run all the "PASSING" tests when --rerun flag is passed instead of only the failed ones.
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/57
Reviewed By: krallin
Differential Revision: D23742159
Pulled By: lukaspiatkowski
fbshipit-source-id: 3b5e89ad29c753d585c1a6f01a9a1d6c1e616fbf
Summary: fixes build and test errors on OSS introduced by D23596262 (deb57a25ed)
Reviewed By: ikostia
Differential Revision: D23757086
fbshipit-source-id: 7973ce36b3589cbe21590bd7e19a9828be72128f
Summary: Since repo_import tool is automated, we need a way to recover the process when the tool break without restarting the whole process. To do this, I defined a new struct (RecoveryFields) that allows us to keep track of the state. The most important fields are the import_stage (ImportStage) we need for keeping track of the process and to indicate the first stage of recovery, and the cs_ids we use throughout the process. For each stage in importing, we save the state after we have finished that part. This way we can also recover from interrupts. To do process recovery, we only need to use `recover-process <file_path>` subcommand, where file_path stores the saved state of importing. For normal run we will use `import [<args>]` subcommand.
Reviewed By: krallin
Differential Revision: D23678367
fbshipit-source-id: c0e0b270ea2ccc499368e54f37550cfa58c03970
Summary:
This change allows us to use warm bookmark cache for all clients except
for external sync job (i.e. the job we use to keep configerator-hg in sync with
configerator-git).
This is useful we'd like to use warm bookmark cache for configerator but it
doesn't work with external sync job. We'd like to use it because warm bookmark
cache doesn't advance a bookmark until this revision showed up in
configerator-hg - this proved to be useful when rolling out configerator for
devservers since there were tools that talked to hg, and they were failing if
hg was behind.
Currently hg external sync job doesn't work with warm bookmark cache because it
tries to incorrect move a master. What I mean by that is that hg external sync
job sends unbundle request which contains a pushkey part which says "move
master from commit A to commit B". If commit A is outdated because of warm
bookmark cache then this update will just fail, because master bookmark
actually points to commit C.
Let's just not ever use warm bookmark cache for external sync job
Reviewed By: aslpavel
Differential Revision: D23754603
fbshipit-source-id: c8eec54bca2224688d4a829ded372c6fc4d7930f
Summary:
Pass the elements to the hasher to avoid needing to alloc a vec from them.
This saves building the vec inside MPathElement, and when used on top of smallvec based MPathElement also saves allocation of a Vec from the SmallVec for each element.
Reviewed By: aslpavel
Differential Revision: D23703342
fbshipit-source-id: dd81c6d69b90f128d697ba847dde34058ad1ea6e
Summary:
Use smallvec for the internal storage of MPathElement.
The previous Bytes had a stack size of 32 bytes plus the text it pointed to.
Using SmallVec we can store up to 24 bytes without allocation keeping the same space as the previous Bytes object.
Given most path elements are directory names and directory names are usually short it is expected that this will save both space and allocations.
Reviewed By: farnz
Differential Revision: D23703344
fbshipit-source-id: 39ffc3bd3bb765bd1dbb757b4b1a7782382db909
Summary:
When sending trees and files we try to avoid sending trees that are
available from the main server. To do so, we currently check to see if the
tree/file is from the local store (i.e. .hg/store instead of $HGCACHE).
In a future diff we'll be moving trees to use the Rust store, which doesn't
expose the difference between shared and local stores. So we need to stop
depending on logic to test the local store.
Instead we can test if the commit is public or not, and only send the tree/file
is the commit is not public. This is technically a revert of the 2018 D7992502 (5e95b0e32e)
diff, which stopped depending on phases because we'd receive public commits from
svn there were not public on the server yet. Since svn is gone, I think it's
safe go back to that way.
This code was usually to help when the client was further ahead than another
client and in some commit cloud edge cases, but 1) we don't do much/any p2p
exchange anymore, and 2) we did some work this year to ensure clients have more
up-to-date remote bookmarks during exchange (as a way of making phases and
discovery more reliable), so hopefully we can rely on phases more now.
Reviewed By: quark-zju
Differential Revision: D23639017
fbshipit-source-id: 34c13aa2b5ef728ea53ffe692081ef443e7e57b8
Summary:
There was a bug. If an entry was skipped then we haven't updated the counter.
That means we might skip the same entry over and over again.
Let's fix it
Reviewed By: ikostia
Differential Revision: D23728790
fbshipit-source-id: f323d14c4deba5736ceb8ada7cb7ee48a69c1272
Summary:
Previously we were able to add a backpressure to the x-repo-sync job that waits
until backsync queue gets empty. However it's also useful to wait until hg sync
queue drains for the large repo. This diff makes it possible to do so.
Reviewed By: aslpavel
Differential Revision: D23728201
fbshipit-source-id: 6b198c8d9c35179169a46f2b804f83838edeff1e
Summary: For fsnodes output the filecontent child nodes first as they can be drained without expanding to more nodes.
Reviewed By: farnz
Differential Revision: D23702268
fbshipit-source-id: 26aeca20d47030dbb9145d406db885fe0fce932c
Summary: Use sorted_vector_map when parsing fsnodes, as inputs are stored sorted, which can result in high cost of BTree insertion when traversing large repos.
Reviewed By: aslpavel
Differential Revision: D23691500
fbshipit-source-id: 1f7a5faf2ef3cb4a72a635d0d8e89037bf4d96b3
Summary:
We are currently logging only the outermost underlying error or context, not any of the lower level causes. This makes mononoke_blobstore_trace less useful!
This changes to use anyhow's alternate format that includes causes
Reviewed By: krallin
Differential Revision: D23708577
fbshipit-source-id: fa2e71734841e2b75d824c456dccf61c1fb13fd2
Summary:
Instead of using force_set and force_delete let's use create() update() and
delete() calls.
Reviewed By: ikostia
Differential Revision: D23704245
fbshipit-source-id: 40bcfd906c4f61a860e5ec8312cddc0d80ea94ae
Summary:
This is the function that was used in repo_import tool to wait until hg sync
has processed all of the entries in the queue. Let's move it to the hg sync
helper lib so that it can be used in other places. E.g. I'd like to use it in
the next diffs in mononoke_x_repo_sync_job.
Reviewed By: krallin
Differential Revision: D23708280
fbshipit-source-id: ea846081d89b55b0d2f5407c971e13869cedfd8b
Summary:
In preparation of moving away from SSH as an intermediate entry point for
Mononoke, let Mononoke work with newly introduced Metadata. This removes any
assumptions we now make about how certain data is presented to us, making the
current "ssh preamble" no longer central.
Metadata is primarily based around identities and provides some
backwards-compatible entry points to make sure we can satisfy downstream
consumers of commits like hooks and logs.
Simarly we now do our own reverse DNS resolving instead of relying on what's
been provided by the client. This is done in an async matter and we don't rely
on the result, so Mononoke can keep functioning in case DNS is offline.
Reviewed By: farnz
Differential Revision: D23596262
fbshipit-source-id: 3a4e97a429b13bae76ae1cdf428de0246e684a27
Summary: Support for multiple heads in `BonsaiDerived::find_all_underived_ancestors`. This change will be needed to remove manual step of fetching of all changesets in `backfill_derived_data` utilty.
Reviewed By: StanislavGlebik
Differential Revision: D23705295
fbshipit-source-id: 32aa97a77f0a4461cbe4bf1864477e3e121e1879
Summary:
As it says in the title, this adds support for receiving compressed responses
in the revisionstore LFS client. This is controlled by a flag, which I'll
roll out through dynamicconfig.
The hope is that this should greatly improve our throughput to corp, where
our bandwidth is fairly scarce.
Reviewed By: StanislavGlebik
Differential Revision: D23652306
fbshipit-source-id: 53bf86d194657564bc3bd532e1a62208d39666df
Summary:
This adds support for compressing responses in the LFS Server, based on what
the client sent in `Accept-Encoding`. The compression changes are fairly
simple. Most of the codes changes are around the fact that when we compress,
we don't send a Content-Length (because we don't know how long the content will
be).
Note that this is largely implemented in StreamBody. This means it can be used
for free by the EdenAPI server as well. The reason it's in there is because we
need to avoid setting the Content-Length when compression is going to be used
(`StreamBody` is what takes charge for doing this). This also exposes a
callback to get access to the stream post-compression, which also needs to be
exposed in `StreamBody`, since that's where compression happens.
Reviewed By: aslpavel
Differential Revision: D23652334
fbshipit-source-id: 8f462d69139991c3e1d37f392d448904206ec0d2
Summary:
This imports the async-compression crate. We have an equivalent-ish in
common/rust, but it targets Tokio 0.1, whereas this community-supported crate
targets Tokio 0.2 (it offers a richer API, notably in the sense that we
can use it for Streams, whereas the async-compression crate we have is only for
AsyncWrite).
In the immediate term, I'd like to use this for transfer compression in
Mononoke's LFS Server. In the future, we might also use it in Mononoke where we
currently use our own async compression crate when all that stuff moves to
Tokio 0.2.
Finally, this also updates zstd: the version we link to from tp2 is actually
zstd 1.4.5, so it's a good idea to just get the same version of the zstd crate.
The zstd crate doesn't keep a great changelog, so it's hard to tell what has changed.
At a glance, it looks like the answer is not much, but I'm going to look to Sandcastle
to root out potential issues here.
Reviewed By: StanislavGlebik
Differential Revision: D23652335
fbshipit-source-id: e250cef7a52d640bbbcccd72448fd2d4f548a48a
Summary: New type async/await can mutate variables, we no longer need synchronization for this counters
Reviewed By: ikostia
Differential Revision: D23704765
fbshipit-source-id: eb2341cb0c82b8a49c28ad3c8fd811ed3af73436
Summary:
This would let us allow only a certain bookmarks to be remapped from a small
repo to a large repo.
Reviewed By: krallin
Differential Revision: D23701341
fbshipit-source-id: cf17a1a21b7594a94c5fb117065f7d9298c8d1af
Summary:
Previously we used target repo for a commit from a source repo. This diff fixes
it.
Reviewed By: krallin
Differential Revision: D23685171
fbshipit-source-id: 4aa105aec244ebcff92b7b71a6cb22dd8a10d2e5
Summary: Add a test to detect any unexpected changes in MPatheElements size
Reviewed By: farnz
Differential Revision: D23703345
fbshipit-source-id: 74354f0861b048ee4611304fc99f0289bce4a7a5
Summary:
Facebook
We need them since we are going to sync ovrsource commits into fbsource
Reviewed By: krallin
Differential Revision: D23701667
fbshipit-source-id: 61db00c7205d5d4047a4040992e7195f769005d3
