Summary:
Correctly identify infinitepush without bookmarks as infinitepush instead of plain push.
Current behavior would sometimes pass `infinitepush` bundles through the `push` pipeline. Interestingly, this does not result in any user-visible effects at the moment. However, in the future we may want to diverge these pipelines:
- maybe we want to disable `push`, but enable `infinitepush`
- maybe there will be performance optimizations, applicable only to infinitepush
In any case, the fact that things worked so far is a consequence of a historical accident, and we may not want to keep it this way. Let's have correct identification.
Reviewed By: StanislavGlebik
Differential Revision: D18934696
fbshipit-source-id: 69650ca2a83a83e2e491f60398a4e03fe8d6b5fe
Summary: This is useful during invesigations. ODS only works for stats.
Reviewed By: krallin
Differential Revision: D21144414
fbshipit-source-id: 0fbb95a79c324d270c8d6dc4770d7729c7b23694
Summary:
In getbundle, we compute the set of new draft commit ids. This is used to
include tree and file data in the bundle when draft commits are fully hydrated,
and will also be used to compute the set of mutation information we will
return.
Currently this calculation only computes the non-common draft heads. It
excludes all of the ancestors, which should be included. This is because it
re-uses the prepare_phases code, which doesn't quite do what we want.
Instead, separate out these calculations into two functions:
* `find_new_draft_commits_and_public_roots` finds the draft heads
and their ancestors that are not in the common set, as well as the
public roots the draft commits are based on.
* `find_phase_heads` finds and generates phase head information for
the public heads, draft heads, and the nearest public ancestors of the
draft heads.
Reviewed By: StanislavGlebik
Differential Revision: D20871337
fbshipit-source-id: 2f5804253b8b4f16b649d737f158fce2a5102002
Summary:
We should use the HgsqlName to check the repo lock, because that's the one
Mercurial uses in the repo lock there.
Reviewed By: farnz
Differential Revision: D20943177
fbshipit-source-id: 047be6cb31da3ee006c9bedc3de21d655a4c2677
Summary: Not in use any more - all hooks are now Bonsai form - so remove it.
Reviewed By: krallin
Differential Revision: D20891164
fbshipit-source-id: b92f169a0ec3a4832f8e9ec8dc9696ce81f7edb3
Summary: This allows the client to do proper feature detection.
Reviewed By: krallin
Differential Revision: D20910379
fbshipit-source-id: c7b9d4073e94518835b39809caf8b068f70cbc2f
Summary:
The Mercurial SHA1 is defined as:
sorted([p1, p2]) + content
The client wants to be able to verify the commit hashes returned by
getcommitdata. Therefore, also write the sorted parents so the client can
calculate the SHA1 easily without fetching SHA1s of parents. This is
useful because we also want to make commit SHA1s lazy on client-side.
I also changed the NULL behavior so the server does not return
content for the NULL commit, as it will fail the SHA1 check.
The server will expects the client to already know how to handle
the NULL special case.
Reviewed By: krallin
Differential Revision: D20910380
fbshipit-source-id: 4a9fb8ef705e93c759443b915dfa67d03edaf047
Summary:
If a blob is redacted, we shouldn't crash in batch. Instead, we should return
that the blob exists, and let the download path return to the client the
information that the blob is redacted. This diff does that.
Reviewed By: HarveyHunt
Differential Revision: D20897247
fbshipit-source-id: 3f305dfd9de4ac6a749a9eaedce101f594284d16
Summary: Running on Mercurial hooks isn't scalable long term - move the consumers of hooks to run on both forms for a transition period
Reviewed By: krallin
Differential Revision: D20879136
fbshipit-source-id: 4630cafaebbf6a26aa6ba92bd8d53794a1d1c058
Summary: We want all hooks to run against the Bonsai form, not a Mercurial form. Create a second form of hooks (currently not used) which acts on Bonsai hooks. Later diffs in the stack will move us over to Bonsai only, and remove support for Mercurial changeset derived hooks
Reviewed By: krallin
Differential Revision: D20604846
fbshipit-source-id: 61eece8bc4ec5dcc262059c19a434d5966a8d550
Summary: As it says in the title!
Reviewed By: HarveyHunt
Differential Revision: D20869828
fbshipit-source-id: df7728ce548739ef2dadad1629817fb56c166b66
Summary:
We use the logged arguments directly for wireproto replay, and then we replay
this directly in traffic replay, but just joining a list with `,` doesn't
actually work for directories:
- We need trailing commas
- We need wireproto encoding
This does that. It also clarifies that this encoding is for debug purposes by
updating function names, and relaxes a bunch of types (since hgproto uses
bytes_old).
Reviewed By: StanislavGlebik
Differential Revision: D20868630
fbshipit-source-id: 3b805c83505aefecd639d4d2375e0aa9e3c73ab9
Summary:
Combined with the unbundle resolver stats, we will be able to say which
percentage of pushrebases fails, for example.
Reviewed By: StanislavGlebik
Differential Revision: D20818224
fbshipit-source-id: 70888b1cb90ffae8b11984bb024ec1db0e0542f7
Summary:
We need this to be able to monitor how frequently we get pushes vs
infinitepushes, etc. A furhter diff will add a similar reporting to
`processing.rs`, so that we can compute a percentage of successful pushes to
all pushes, for example.
Reviewed By: StanislavGlebik
Differential Revision: D20818225
fbshipit-source-id: 7945dc285560d1357bdc6aef8e5fe50b61622254
Summary:
Migrate the configuration of sql data managers from the old configuration using `sql_ext::SqlConstructors` to the new configuration using `sql_construct::SqlConstruct`.
In the old configuration, sharded filenodes were included in the configuration of remote databases, even when that made no sense:
```
[storage.db.remote]
db_address = "main_database"
sharded_filenodes = { shard_map = "sharded_database", shard_num = 100 }
[storage.blobstore.multiplexed]
queue_db = { remote = {
db_address = "queue_database",
sharded_filenodes = { shard_map = "valid_config_but_meaningless", shard_num = 100 }
}
```
This change separates out:
* **DatabaseConfig**, which describes a single local or remote connection to a database, used in configuration like the queue database.
* **MetadataDatabaseConfig**, which describes the multiple databases used for repo metadata.
**MetadataDatabaseConfig** is either:
* **Local**, which is a local sqlite database, the same as for **DatabaseConfig**; or
* **Remote**, which contains:
* `primary`, the database used for main metadata.
* `filenodes`, the database used for filenodes, which may be sharded or unsharded.
More fields can be added to **RemoteMetadataDatabaseConfig** when we want to add new databases.
New configuration looks like:
```
[storage.metadata.remote]
primary = { db_address = "main_database" }
filenodes = { sharded = { shard_map = "sharded_database", shard_num = 100 } }
[storage.blobstore.multiplexed]
queue_db = { remote = { db_address = "queue_database" } }
```
The `sql_construct` crate facilitates this by providing the following traits:
* **SqlConstruct** defines the basic rules for construction, and allows construction based on a local sqlite database.
* **SqlShardedConstruct** defines the basic rules for construction based on sharded databases.
* **FbSqlConstruct** and **FbShardedSqlConstruct** allow construction based on unsharded and sharded remote databases on Facebook infra.
* **SqlConstructFromDatabaseConfig** allows construction based on the database defined in **DatabaseConfig**.
* **SqlConstructFromMetadataDatabaseConfig** allows construction based on the appropriate database defined in **MetadataDatabaseConfig**.
* **SqlShardableConstructFromMetadataDatabaseConfig** allows construction based on the appropriate shardable databases defined in **MetadataDatabaseConfig**.
Sql database managers should implement:
* **SqlConstruct** in order to define how to construct an unsharded instance from a single set of `SqlConnections`.
* **SqlShardedConstruct**, if they are shardable, in order to define how to construct a sharded instance.
* If the database is part of the repository metadata database config, either of:
* **SqlConstructFromMetadataDatabaseConfig** if they are not shardable. By default they will use the primary metadata database, but this can be overridden by implementing `remote_database_config`.
* **SqlShardableConstructFromMetadataDatabaseConfig** if they are shardable. They must implement `remote_database_config` to specify where to get the sharded or unsharded configuration from.
Reviewed By: StanislavGlebik
Differential Revision: D20734883
fbshipit-source-id: bb2f4cb3806edad2bbd54a47558a164e3190c5d1
Summary:
For the initial rollout of lfs on fbsource we want to rollout just for our
team using rollout_smc_tier option. This diff adds a support for that in
Mononoke.
It spawns a future that periodically updates list of enabled hosts in smc tier.
I had a slight concern about listing all the available services and storing
them in memory - what if smc tier have too many services? I decided to go ahead
with that because
1) [Smc antipatterns](https://fburl.com/wiki/ox43ni3a) wiki page doesn't seem
to list it as a concern.
2) We are unlikely to use for large tier - most likely we'll use it just for
hg-dev which contains < 100 hosts.
Reviewed By: krallin
Differential Revision: D20789751
fbshipit-source-id: d35323e49530df6983e159e2ed5bce205cc5666d
Summary:
We are currently having problems with streaming clone:
```
$ hg --config 'extensions.fsmonitor=!' clone --shallow -U --config 'ui.ssh=ssh -oControlMaster=no' --configfile /etc/mercurial/repo-specific/fbsource.rc 'ssh://hg.vip.facebook.com//data/scm/fbsource?force_mononoke' "$(pwd)/fbsource-clone-test"
remote: server: https://fburl.com/mononoke
remote: session: vJ3qkiQIm9FT7mCp
connected to twshared11499.02.cln2.facebook.com
streaming all changes
2 files to transfer, 5.42 GB of data
abort: unexpected response from remote server:
'\x00\x01B?AB\x00\x00\x00\x00\x02U\x00\x00\x02\xc7\x00b\xf0\xd5\x00b\xf0\xd5\x00b\xf0\xd4\xff\xff\xff\xff\xa8z\xc7W\xd0&\xab\xb2\xf1{\xbfq\xac<\xaf6W\x06q\x81\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01B?C\x97\x00\x00\x00\x00\x053\x00\x00\x06\xce\x00b\xf0\xd6\x00b\xf0\xd6\x00b\xf0\xd5\xff\xff\xff\xff\xa3I\x19+\xe2\x0f\xae\xd2\x95\x14\x8a\xde\x19\x18\xf0\x8cUQu\xf1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01B?H\xca\x00\x00\x00\x00\x02\xe4\x00\x00\x03\x9e\x00b\xf0\xd7\x00b\xf0\xd7\x00b\xf0\xd6\xff\xff\xff\xffx\xd6}\x12nt\xb9\xbc(\x83\xfb\xfa\xcc\xc1o?\xde\xcc\x06L\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01B?K\xae\x00\x00\x00\x00\x02j\x00\x00\x02\xb5\x00b\xf0\xd8\x00b\xf0\xd8\x00b\xf0\xd7\xff\xff\xff\xff\x04"\xfcw6\'M\xba\xf1f\xdb\x02\xbeE\x93:\xc8\x17\x88P\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01B?N\x18\x00\x00\x00\x00\x03\xbb\x00\x00\x04\xb8\x00b\xf0\xd9\x00b\xf0\xd9\x00b\xf0\xd8\xff\xff\xff\xff\xb9\x15*p/\xa4*\x00\x9dZw\x01B\x87L\x8f\x08\x11\x89\xe0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0000changelog.d\x005406413267\n'
```
as the result of the debugging it is turned out that we are sending more data than expected, to have better error next time if we have any corruption of the `streaming_changelog_chunks` table
Reviewed By: StanislavGlebik
Differential Revision: D20763738
fbshipit-source-id: 6f6fa9f9a29909e044d9ba42fe84916ddcb62e8f
Summary:
As suggested in D20680173, we can reduce the overall need to copy things by
storing refs in the resolver.
Reviewed By: krallin
Differential Revision: D20696588
fbshipit-source-id: 9456e2e208cfef6faed57fc52ca59fafdccfc68c
Summary:
See bottom diff of this stack for overview.
This diff in particular asyncifies the `upload_changeset` fn. Apart from that,
it also makes sure it can accept `&RevlogChangeset` instead of
`RevlogChangeset`, which helps us to get rid of cloning.
Reviewed By: krallin
Differential Revision: D20693932
fbshipit-source-id: b0e5e1604cbfb6f6b6e269c85a79208115325734
Summary: Same as the bottom diff of this stack, but for another file.
Reviewed By: krallin
Differential Revision: D20693934
fbshipit-source-id: 4c2d12bf9d9ab272898a7830ece6d9f563adb8fb
Summary:
This diff focuses on the following:
- replaces clones with references, both when this decreases the total sum of
clones, and when it causes the only clone to be on the boundary with the
compat code. This, when those boundaries are pushed further, we can only fix
one place in resolver
- removes a weird wrapping of a closure into an `Arc` and just calls
`upload_changesets` directly instead
- in cases when `BundleResolver` methods take `ctx` as an argument removes it
and makes those methods use the one stored in the struct
Reviewed By: StanislavGlebik
Differential Revision: D20680173
fbshipit-source-id: c397c4ade57a07cbbc9206fa8a44f4225426778c
Summary:
This updates unbundle_replay to account for pushrebase hooks, notably to assign
globalrevs.
To do so, I've extracted the creation of pushrebase hooks in repo_client and
reused it in unbundle_replay. I also had to update unbundle_replay to no longer
use `args::get_repo` since that doesn't give us access to the config (which we
need to know what pushrebase hooks to enable).
Reviewed By: ikostia
Differential Revision: D20622723
fbshipit-source-id: c74068c920822ac9d25e86289a28eeb0568768fc
Summary:
This adds a unbundle_replay Rust binary. Conceptually, this is similar to the
old unbundle replay Python script we used to have, but there are a few
important differences:
- It runs fully in-process, as opposed to pushing to a Mononoke host.
- It will validate that the pushrebase being produced is consistent with what
is expected before moving the bookmark.
- It can find sources to replay from the bookmarks update log (which is
convenient for testing).
Basically, this is to writes and to the old unbundle replay mechanism what
Fastreplay is to reads and to the traffic replay script.
There is still a bit of work to do here, notably:
- Make it possible to run this in a loop to ingest updates iteratively.
- Run hooks.
- Log to Scuba!
- Add the necessary hooks (notably globalrevs)
- Set up pushrebase flags.
I would also like to see if we can disable the presence cache here, which would
let us also use this as a framework for benchmarking work on push performance,
if / when we need that.
Reviewed By: StanislavGlebik
Differential Revision: D20603306
fbshipit-source-id: 187c228832fc81bdd30f3288021bba12f5aca69c
Summary:
A few of our tasks failed on startup and most likely it was during warmup
though we are not sure (see attached task).
Let's add move logging
Reviewed By: farnz
Differential Revision: D20698273
fbshipit-source-id: 4facd21a94d2917103e417a014b820c893da4718
Summary:
See bottom diff of the stack for overview.
This diff asyncifies the resolver's entrypoint: `resolve` fn, and provides a compatibility shim `resolve_compat` to call from old-style code. Alternatively, we could just do `async move { resolve(...).await }.boxed().compat()` at a callsite. This did not seem too important to do one way or the other, let me know if you feel strongly about it.
Reviewed By: StanislavGlebik
Differential Revision: D20634406
fbshipit-source-id: ee3c73a7a2c65c333e95194bd90ca7330b225011
Summary: See bottom diff of the stack for overview.
Reviewed By: StanislavGlebik
Differential Revision: D20633227
fbshipit-source-id: 16b3f3a764a75261da0585c9724a17853e865681
Summary:
See bottom diff of the stack for overview.
This diff asyncifies `resolve_push`.
Reviewed By: StanislavGlebik
Differential Revision: D20620100
fbshipit-source-id: 1109933e388d9485f42c63638621a7b9227f157f
Summary:
See bottom diff of the stack for overview.
This diff asyncifies `resolve_bookmark_only_pushrebase`.
Reviewed By: StanislavGlebik
Differential Revision: D20610253
fbshipit-source-id: 2a79ac9e8bdca18401ed95d98a0e1b3e92fee4fe
Summary:
See bottom diff of the stack for overview.
This diff asyncifies `return_with_rest_of_bundle`.
Reviewed By: StanislavGlebik
Differential Revision: D20605807
fbshipit-source-id: 0df7d18d06720ff166cdc3e9b981932a819cb0aa
Summary:
See bottom diff of the stack for overview.
This diff asyncifies `next_item`
Reviewed By: StanislavGlebik
Differential Revision: D20605817
fbshipit-source-id: 99fc0100736f0a9307448c6f2ead91da81a531cb
Summary:
See the bottom diff of the stack for overview.
This diff asyncifies two fns: `maybe_save_full_content_bundle2 ` and `is_next_part_pushkey`
Reviewed By: StanislavGlebik
Differential Revision: D20605814
fbshipit-source-id: 1daea901e7620638fa9b8d0b69c18b0ff4e967da
Summary:
See bottom diff of the stack for overview.
This diff asyncifies `maybe_resolve_commonheads` fn.
Reviewed By: StanislavGlebik
Differential Revision: D20605808
fbshipit-source-id: 6b628a4c66970e468839732db8ffb11c961be591
Summary:
See bottom diff of the stack for overview.
This diff asyncifies `maybe_resolve_pushvars` fn.
Reviewed By: StanislavGlebik
Differential Revision: D20605812
fbshipit-source-id: e68f9d878a294a9980e53b104aa1035c0d47ae65
Summary:
See bottom diff of the stack for overview.
This diff asyncifies `maybe_resolve_changegroup`.
Reviewed By: StanislavGlebik
Differential Revision: D20605810
fbshipit-source-id: fbf5e9d93b355dfa23a3a7657edb96b033535f9d
Summary:
See bottom diff of the stack for overview.
This diff asyncifies `ensure_stream_finished` fn.
Reviewed By: StanislavGlebik
Differential Revision: D20605805
fbshipit-source-id: 853a6d5d1afeee0f8f841eec51302fb3ceb701c7
Summary:
See bottom diff of the stack for motivation.
This diff asyncifies `resolver_multiple_parts` and `maybe_resolve_pushkey` functions.
Reviewed By: StanislavGlebik
Differential Revision: D20605815
fbshipit-source-id: 90768f4495632ec83b79ca9fcf982b0ec5c277cf
Summary:
See bottom diff of the stack for overview.
This diff asyncifies `resolve_b2xtreegroup2`.
Reviewed By: StanislavGlebik
Differential Revision: D20605806
fbshipit-source-id: 2e667d19f2014d051d25a74353e9ebd2e6a93c72
Summary:
See bottom diff of the stack for the overview.
This particular diff asyncifies the `maybe_resolve_infinitepush_bookmarks` fn.
Reviewed By: StanislavGlebik
Differential Revision: D20605816
fbshipit-source-id: 11b6e9c5dd7423bcc4ecc988efd581a3e970ccdc
Summary:
See the bottom diff of the stack for the overview.
This diff specifically migrates the `upload_changesets` function.
Reviewed By: StanislavGlebik
Differential Revision: D20605809
fbshipit-source-id: 36a11a72fb828d494bd18c7737e2682cb3b7cb9a
Summary:
See D20605813 for the overview of the stack.
This diff migrates a few leaf functions.
Reviewed By: krallin
Differential Revision: D20605811
fbshipit-source-id: 2f5d5e5fba3a00afd61a4eb58c505658ac82943a
Summary:
A wider goal of this stack is to migrate `repo_client/unbundle/src/resolver.rs` to async/await and new futures.
The approach is as follows:
- rename old futures upon import into `OldFuture`, `OldBoxFuture`, etc. so that it is easily visible where we use what [this diff]
- implement a bunch of mechanical conversions of continuation-passing-style code to the imperative async/await based code without worrying about clones-vs-references or excessive boxification. Keep individual diffs as small and as mechanical as possible, so that it is easier to review.
- once the `resolve` fn is migrated, introduce `resolve_compat`, which is used from `repo_client/client/src/mod.rs`
- then go through the codebase and see where we can remove clones of resolver/ctx and excessive `boxed()/boxify()` if any are left
Note: `Bundle2` is an `OldStream` and I postpone its migration till the high-level structure of the `resolver.rs` is migrated, since the main value is in allowing imperative-style code in the file
Reviewed By: krallin
Differential Revision: D20605813
fbshipit-source-id: 32255d7b3573f87f74a496e6e40b842e553242a7
Summary:
Run buck build -c rust.clippy=true eden/mononoke/:mononoke#check and fix some
of them manually. I wasn't able to make rustfix to work - will try to see
what's wrong and run it.
The suggestions looks non-controversial
Reviewed By: krallin
Differential Revision: D20520123
fbshipit-source-id: 25d4eb493f2363c5aa77bdb3876da4378483f6cb
Summary: 'new' is not very explicit with the fact that things are not refreshed.
Reviewed By: dtolnay
Differential Revision: D20356129
fbshipit-source-id: ff4a8c6fe4c34e93729c902e4b41afbe3c9deca1
