Summary:
Do a similar change to change_target_config as we've done for add_sync_target
in D29848378. Move bookmark only if it points to an expected commit. That would
prevent make it safer to deal with cases where the same change_target_config
was executing twice.
Reviewed By: mojsarn
Differential Revision: D29874803
fbshipit-source-id: d21a3029ee58e2a8acc41e37284d0dd03d2803a3
Summary:
This is the first diff that tries to make megarepo asynchronous methods
idempotent - replaying the same reqeust twice shouldn't cause corruption on the
server. At the moment this is not the case - if we have a runaway
add_sync_target call, then in the end it moves a bookmark to a random place,
even if there was another same successful add_sync_target call and a few others on
top.
add_sync_target should create a new bookmark, and if a bookmark already exists
it's better to not move it to a random place.
This diff does it, however it creates another problem - if a request was successful on mononoke side, but we failed to deliver the successful result to the client (e.g. network issues), then retrying this request would fail because bookmark already exists. This problem will be addressed in the next diff.
Reviewed By: mojsarn
Differential Revision: D29848378
fbshipit-source-id: 8a58e35c26b989a7cbd4d4ac4cbae1691f6e9246
Summary: As discussed, extends Mononoke service to support commits w/o parents for the AI Infra usecase.
Reviewed By: markbt
Differential Revision: D29810303
fbshipit-source-id: f07fd7f1521ffe1cea85f1f54e71fe37fc39bb62
Summary: It's nice to be able to keep track of what's going on
Reviewed By: mwdevine
Differential Revision: D29790543
fbshipit-source-id: b855d72efe8826a99b3a6a562722e299e9cbfece
Summary:
Added an optional argument to `/upload/file`, that allows specifying a bubble id, which will be used to upload the file into the ephemeral blobstore instead of the main one.
This is necessary in order to create a snapshot, as all files must be in the ephemeral blobstore.
Reviewed By: liubov-dmitrieva
Differential Revision: D29734333
fbshipit-source-id: c1dcf8d5a78819925f8defbfbd7d06b0f6a9e973
Summary: Insert id's are always positive, so let's use `NonZeroU64` instead of `u64`. This is more restricted, which is good, but also has the added benefit that `Option<NonZeroU64>` doesn't use any additional space, because of compiler optimizations.
Reviewed By: StanislavGlebik
Differential Revision: D29733877
fbshipit-source-id: 8a0e1a1bd84bcedbba51840f1da8f8cac79bca42
Summary: Ephemeral handle is a blobstore that's built from a bubble and a "main blobstore", which first attempts to read from the ephemeral blobstore, but falls back to the main one. Will be used to read/write stuff in snapshots.
Reviewed By: liubov-dmitrieva
Differential Revision: D29733408
fbshipit-source-id: f15ae9d3009632cd71fafa88eac09986e0b958e7
Summary:
Move EdenApi Uploads code from commit cloud extension to core
So this can be later used for pushes as well. The code is not commit cloud specific.
The function takes revs and returns uploaded, failed lists that are also revs.
Reviewed By: yancouto
Differential Revision: D29846299
fbshipit-source-id: e3a7fbc56f0b651c738dc06da7fdb7cde4feedf7
Summary: This test is overly reliant on exact logging output, and the output has changed. Update the test for the new output, and make it a bit more lenient in the process.
Reviewed By: StanislavGlebik
Differential Revision: D29787827
fbshipit-source-id: 3e8aa77d2edcf3d0ca95c0d17d0b4e3845b78ae3
Summary:
Improve integration tests coverage for `hg cloud upload` and `hg cloud sync` with enabled upload.
This includes end2end tests for uploading mutation information, pulling commits from another repo,
generally how uploads behaves after a rebase, after file moves, after editing a commit message, how copy_from data has been preserved.
Reviewed By: markbt
Differential Revision: D29816436
fbshipit-source-id: 2aa421c8479683721984e13d537c34df8b1ca2d1
Summary: update test certificates for another 10 years rather than the default 1 year
Reviewed By: markbt
Differential Revision: D29846930
fbshipit-source-id: 98bc139c21e4d9e4cb5bab46485d849345bcc43d
Summary:
Add methods to easily determine whether a tree exists, or whether anything
(either a file or a tree) exists at a particular path.
Reviewed By: StanislavGlebik
Differential Revision: D29815982
fbshipit-source-id: f3fb1919545bdcb46ed663a0a514338dc137abee
Summary:
This is not used. Even though this method has the "right intention" (i.e. we
need to start marking long running requests as new), I'm not sure we can use it
as is. So let's just delete it for now.
Reviewed By: farnz
Differential Revision: D29817068
fbshipit-source-id: 84d392fea01dfb5fb7bc56f0072baf2cf70b39f4
Summary: Currently we only print what's in the `error` annotation.
Reviewed By: krallin
Differential Revision: D29794843
fbshipit-source-id: a2c411208d7be8fd856dd9b3f82fd96a4ed37aee
Summary:
bugfix: fix uninitalized state variable and add a test
in rare cases it is used further down the code
Reviewed By: StanislavGlebik
Differential Revision: D29815203
fbshipit-source-id: e117df5575f025787d94f0a8ed4a171408e361d0
Summary:
We seem to be reloading it every minute, even though we are supposed to reload
only when it's changed. That's probably not a huge deal, but we just get a
spammy stderr message. Let's remove it.
Reviewed By: yancouto
Differential Revision: D29789760
fbshipit-source-id: 65a39cca67636ae71befb963c78b6473b5b9f3fc
Summary:
mysql tests were failing because of invalid config with
```
+ E0719 14:48:27.582197 1846476 [main] eden/mononoke/cmdlib/src/helpers.rs:318] Execution error: unknown keys in config parsing: `{"blobstore.ephemeral_blobstore.?.metadata.?.filenodes", "blobstore.ephemeral_blobstore.?.metadata.?.mutation", "blobstore.ephemeral_blobstore.?.metadata.?.primary"}`
```
See example - https://www.internalfb.com/intern/testinfra/diagnostics/6473924511735259.562949979040542.1626706163/
This diff fixes it
Reviewed By: akushner
Differential Revision: D29812804
fbshipit-source-id: c71f7f38103194137523ca947e4b23819da37c35
Summary:
Rename to avoid confusion. The function filters errors from the underlying stream.
The first error and number of errors are logged to scuba but the errors are not passed to the client.
Reviewed By: kulshrax
Differential Revision: D29734930
fbshipit-source-id: 503adaa9e618d931a354011ef83c3ab22eb3b9bf
Summary:
Using all the preparations added in the stack, this diff adds the `/:repo/ephemeral/prepare` endpoint to eden api.
It simply creates an ephemeral bubble and returns its id via the call.
Reviewed By: markbt
Differential Revision: D29698714
fbshipit-source-id: 5bc289cad97657db850b151849784e50a17a9da6
Summary: This allows ephemeral blobstore to be used in places that have a Repo context, like in the eden api, which will be used on the next diff to implement a new endpoint on eden api to create a bubble.
Reviewed By: markbt
Differential Revision: D29697657
fbshipit-source-id: b7e83c5c7c5e77243f0dba29c024d9f66ca4b2f9
Summary:
Config for ephemeral blobstore and some code for creating ephemeral blobstores was already added, this diff ties them both together by making the ephemeral blobstore be build using the default config on RepoFactory, so it can be used as a Repo attribute easily in other places.
I was able to do this easily because I stopped using `BackingBlostore` and started simply using `dyn Blobstore` in the ephemeral blobstore. Using BackingBlobstore would require some significant changes, because:
1. Building of blobstores is not ergonomic, it is quite hard and requires a bunch of manual code to be able to build some subtrait of Blobstore.
2. A lot of the blobstores "wrappers" do not implement things like BlobstoreKeySource, which would need to be implemented individually (example: D29678881 (817948ca75) would be just the start).
Reviewed By: markbt
Differential Revision: D29677545
fbshipit-source-id: 0f5cffe6bdfece1aaa74339ef40376d1ff27e6c2
Summary:
Use the class added on previous diff on segmented changelog periodic reloader as well.
To do this, I needed to add some changes to reloader:
- Add auto implementation of `Loader` trait for functions
- Add a tokio notify, as that was used on tests in segmented changelog
Reviewed By: markbt
Differential Revision: D29524220
fbshipit-source-id: 957f21db91f410fcdabb0d1c16d5c4f615892ab6
Summary: If we ever want to start using things like BlobstoreKeySource more extensively, we'll need to implement it for a lot of blobstores. This starts that, though it's not used for now.
Reviewed By: ahornby
Differential Revision: D29678881
fbshipit-source-id: 918a169b8b934c6f5e1eefaba7d11dc220eb7c59
Summary: This is needed to disable sync-queue lookups and second blobstores lookups later on.
Reviewed By: StanislavGlebik
Differential Revision: D29663435
fbshipit-source-id: abb5109de6063158a7ff0a116a5c1d336bfdb43f
Summary: This just helps to understand where we definitely have to fail in case of "ProbablyNotPresent" and work on those in the future.
Reviewed By: StanislavGlebik
Differential Revision: D29663436
fbshipit-source-id: c8428115f3c9637114e3964c948123d473207d53
Summary:
Segmented changelog is initialized in every BlobRepo, and that's quite annoying
- there's a lot of spam goes to stderr in jobs like hg sync job which don't use
segmented changelog at all.
At the same time segmented changelog is only used in mononoke api, so we can
just initialize segmented changelog in InnerRepo, and remove from BlobRepo
completely.
Reviewed By: markbt
Differential Revision: D29735623
fbshipit-source-id: 9137c9266169b7ef16b1c6c0b80cae896214203b
Summary:
Remove everything about the infinitepush write path.
Infinitepush path has been splitted into 2 for migration purpose. It is now time to clean up.
Reviewed By: StanislavGlebik
Differential Revision: D29711414
fbshipit-source-id: c61799fe124e2def4254cdd45e550c82c501e514
Summary:
Now that the new `rate_limiting` crate is being used by LFS server we
can remove the throttle limits code and config.
Differential Revision: D29396505
fbshipit-source-id: 19638bd93ad9dea2638e8501837c6c13e4dd48ff
Summary:
Integration tests rely on specific debug output. This changed for `String`
in Rust 1.53, so update accordingly.
Reviewed By: yancouto
Differential Revision: D29696713
fbshipit-source-id: 751d72660f1d8772d754ab404192281857b32b2f
Summary:
Implement the `content` attribute.
Introduce a new `FileContent` type which stores the hg file blob and metadata, and modify `FileEntry` to allow constructing `FileEntry` with optional `FileContent` builder-style.
Reviewed By: DurhamG
Differential Revision: D29647203
fbshipit-source-id: b956c294d03dc81affc90d7274b2e430a3556e96
Summary:
Add support for optional file attributes to EdenApi, with `content` there as a placeholder.
Modifies the `FileRequest` type, adding a vec of `FileSpec`, which allows the client to specify desired attributes per-key. The existing `keys` field will be treated as a request for the content attribute and may be used in combination with the new per-key attributes.
Reviewed By: DurhamG
Differential Revision: D29634709
fbshipit-source-id: 6571837f87d1635e8529490e10dbe4ba054b7348
Summary:
This was a hack to allow the tests to produce the same hashes as
before. Let's disable this and fix the remaining test failures. A future diff
will remove the feature entirely.
Where possible I changed input hashes to desc() and output hashes to globs so
hopefully future hash changes are a little easier.
Differential Revision: D29567762
fbshipit-source-id: cf5150c112c56b08f583feba80e5a636cc07db0a
Summary:
This was accidentally committed in an earlier diff. It's unused, so
let's delete it.
Reviewed By: krallin
Differential Revision: D29668138
fbshipit-source-id: 105bf466665c447c37c73462e102d8771d0368ee
Summary:
Support exchange of mutation information during changesets uploads
Add new api for mutation information.
Add implementation for this new api.
Add client side support.
Reviewed By: markbt
Differential Revision: D29661255
fbshipit-source-id: 1d8cfa356599c215460aee49dd0c78b11af987b8
Summary:
Allow SCS server to use blame V2 to serve blame requests, if it is enabled.
This uses `CompatBlame` so that it can use either blame V1 or blame V2.
Reviewed By: liubov-dmitrieva
Differential Revision: D29645410
fbshipit-source-id: 8d02e295995439c3b64e0128bdb5e6f5f6153159
Summary:
Allow access to blame V2 in `mononoke_admin` by using `fetch_blame_compat`.
This uses `CompatBlame` to provide blame support using either blame V1 or blame V2.
Reviewed By: liubov-dmitrieva
Differential Revision: D29492859
fbshipit-source-id: 38c73690d36b57be73cec98ae2a013f16b3e0f7a
Summary:
Implement `fetch_blame_compat`, which will fetch either blame V1 or blame V2,
depending on the repo config, and return a compatibility adapter that can be
used by code to use both kinds.
Reviewed By: StanislavGlebik
Differential Revision: D29492857
fbshipit-source-id: 88d68ef2988e316642a5ebd9aa38b541c02c5da4
Summary:
Add `blame_version` to `BlameDeriveOptions`, and if this is set to `V2`, derive
a V2 blame root for the changeset.
Blame V2 and its roots are in a separate blobstore key space, so this derivation
is entirely independent of blame V1.
The key prefix for blame V2 roots is `derived_root_blame_v2`, even though this
is slightly different to the prefix for blame V1. This is so that it matches
other derived data roots (e.g. unode V2). Similarly, `BlameRoot` becomes
`RootBlameV2` so that it matches the other root types.
Blame V2 uses a separate mapping for blame roots, which contain the root
unode manifest id as additional data.
Differential Revision: D29492858
fbshipit-source-id: de2799040129e1ab90cc6bd8f775a6d47c607db7
Summary:
Split the `derived` module into `derive_v1`, which handles derivation of blame
V1, and `mapping_v1`, which handles the derived data mapping.
This is in preparation for introducing derivation of blame V2.
Reviewed By: StanislavGlebik
Differential Revision: D29463127
fbshipit-source-id: ae3add600ca62141e7f25713367680b667507da3
Summary:
Extract `fetch_content_for_blame` to a separate module so we can re-use it in
blame V2.
The method previously returned nested `Result`s, which can be confusing as in
most contexts, the blame being rejected is not actually an error. Switch to an
explicit enum to make it clearer what the inner result represents.
Reviewed By: yancouto
Differential Revision: D29462095
fbshipit-source-id: 52ffcb4173a3b36f4b6cdafe4f42a4cafd993f49
Summary:
Implement using of uploading changesets in `hg cloud upload` command.
This is the last part for `hg cloud upload` - uploading changesets via Edenapi
test
```
```
# machine #2
liubovd {emoji:1f352} ~/fbsource
[15] → hg pull -r 0b6075b4bda143d5212c1525323fb285d96a1afb
pulling from mononoke://mononoke.c2p.facebook.net/fbsource
connected to twshared27150.03.cln3.facebook.com session RaIPDgvF6l8rmXkA
abort: 0b6075b4bda143d5212c1525323fb285d96a1afb not found!
```
```
# machine #1
devvm1006.cln0 {emoji:1f440} ~/fbsource/fbcode/eden/scm
[6] → EDENSCM_LOG="edenapi::client=info" ./hg cloud upload
Jul 11 13:26:26.322 INFO edenapi::client: Requesting lookup for 1 item(s)
commitcloud: head '0b6075b4bda1' hasn't been uploaded yet
Jul 11 13:26:26.472 INFO edenapi::client: Requesting lookup for 6 item(s)
commitcloud: queue 1 commit for upload
Jul 11 13:26:26.648 INFO edenapi::client: Requesting lookup for 1 item(s)
commitcloud: queue 0 files for upload
Jul 11 13:26:26.698 INFO edenapi::client: Requesting lookup for 4 item(s)
commitcloud: queue 4 trees for upload
Jul 11 13:26:27.393 INFO edenapi::client: Requesting trees upload for 4 item(s)
commitcloud: uploaded 4 trees
commitcloud: uploading commit '0b6075b4bda143d5212c1525323fb285d96a1afb'...
Jul 11 13:26:28.426 INFO edenapi::client: Requesting changesets upload for 1 item(s)
commitcloud: uploaded 1 commit
```
```
# machine #2
liubovd {emoji:1f352} ~/fbsource
[16] → hg pull -r 0b6075b4bda143d5212c1525323fb285d96a1afb
pulling from mononoke://mononoke.c2p.facebook.net/fbsource
connected to twshared16001.08.cln2.facebook.com session QCpy1x9yrflRF6xF
searching for changes
adding commits
adding manifests
adding file changes
added 895 commits with 0 changes to 0 files
(running background incremental repack)
prefetching trees for 4 commits
liubovd {emoji:1f352} ~/fbsource
[17] → hg up 0b6075b4bda143d5212c1525323fb285d96a1afb
warning: watchman has recently started (pid 93231) - operation will be slower than usual
connected to twshared32054.08.cln2.facebook.com session Hw91G8kRYzt4c5BV
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
liubovd {emoji:1f352} ~/fbsource
[18] → hg diff -c .
connected to twshared0965.07.cln2.facebook.com session rrYSvRM6pnBYZ2Fn
diff --git a/fbcode/eden/scm/test b/fbcode/eden/scm/test
new file mode 100644
--- /dev/null
+++ b/fbcode/eden/scm/test
@@ -0,0 +1,1 @@
+test
```
Initial perf wins:
Having a large stack of 6 commits (total 24 files changed), tested *adding a single line to a file at the top commit*. We can see at least 2X win but it should be more because I have tested with a local instance of edenapi service that runs on my devserver.
```
╷
╷ @ 5582fc8ee 6 minutes ago liubovd
╷ │ test
╷ │
╷ o d55f9bb65 86 minutes ago liubovd D29644738
╷ │ [hg] edenapi: Implement using of uploading changesets in `hg cloud upload` command
╷ │
╷ o 561149783 Friday at 15:10 liubovd D29644797
╷ │ [hg] edenapi: Add request handler for uploading hg changesets
╷ │
╷ o c3dda964a Friday at 15:10 liubovd D29644800
╷ │ [edenapi_service] Add new /:repo/upload/changesets endpoint
╷ │
╷ o 28ce2fa0c Friday at 15:10 liubovd D29644799
╷ │ [hg] edenapi/edenapi_service: Add new API for uploading Hg Changesets
╷ │
╷ o 13325b361 Yesterday at 15:23 liubovd D29644798
╭─╯ [edenapi_service] Implement uploading of hg changesets
```
```
# adding new line to a file test in the test commit, and then run:
devvm1006.cln0 {emoji:1f440} ~/fbsource/fbcode/eden/scm
[8] → time hg cloud upload
commitcloud: head '4e4f947d73e6' hasn't been uploaded yet
commitcloud: queue 1 commit for upload
commitcloud: queue 0 files for upload
commitcloud: queue 4 trees for upload
commitcloud: uploaded 4 trees
commitcloud: uploading commit '4e4f947d73e676b63df7c90c4e707d38e6d0a93b'...
commitcloud: uploaded 1 commit
real 0m3.778s
user 0m0.017s
sys 0m0.027s
```
```
# adding another new line to a file test in the test commit, and then run:
devvm1006.cln0 {emoji:1f440} ~/fbsource/fbcode/eden/scm
[11] → time hg cloud backup
connected to twshared30574.02.cln2.facebook.com session uvOvhxtBfeM7pMgl
backing up stack rooted at 13325b3612d2
commitcloud: backed up 1 commit
real 0m7.507s
user 0m0.013s
sys 0m0.030s
```
Test force mode of the new command that reupload everything:
```
devvm1006.cln0 {emoji:1f440} ~/fbsource/fbcode/eden/scm
[13] → time hg cloud upload --force
commitcloud: head '5582fc8ee382' hasn't been uploaded yet
commitcloud: queue 6 commits for upload
commitcloud: queue 24 files for upload
commitcloud: uploaded 24 files
commitcloud: queue 61 trees for upload
commitcloud: uploaded 61 trees
commitcloud: uploading commit '13325b3612d20c176923d1aab8a28383cea2ba9a'...
commitcloud: uploading commit '28ce2fa0c6a02de57cdc732db742fd5c8f2611ad'...
commitcloud: uploading commit 'c3dda964a71b65f01fc4ccadc9429ee887ea982c'...
commitcloud: uploading commit '561149783e2fb5916378fe27757dcc2077049f8c'...
commitcloud: uploading commit 'd55f9bb65a0829b1731baa686cb8a6e0c5500cc2'...
commitcloud: uploading commit '5582fc8ee382c4c367a057db2a1781377bf55ba4'...
commitcloud: uploaded 6 commits
real 0m7.830s
user 0m0.011s
sys 0m0.032s
```
We can see the time is similar to the current `hg cloud backup` command.
Reviewed By: markbt
Differential Revision: D29644738
fbshipit-source-id: cbbfcb2e8018f83f323f447848b3b6045baf47c5
Summary:
implement uploading of hg changesets
For now, reuse the upload code path from unbundle but calling it with empty filenodes and manifests.
Those are used for parents validation but this is not needed for us because we load trees and filenodes and their parents to construct the bonsai cs.
We might want to rewrite it to a cleaner code later and separate from unbundle but for now reusing the fucntion is the easiest way because we know the implementation is correct and also has logging.
Reviewed By: markbt
Differential Revision: D29644798
fbshipit-source-id: 27217d3061ab8d9712417facdbfbbc7e3aebfc5b
Summary: There is a code path that `heads` contain duplicated items. Be compatible with it.
Reviewed By: andll
Differential Revision: D29645743
fbshipit-source-id: ff73bc51e877c2d02fcfff28cd1001e70478f212
Summary:
In previous diff we started creating deletion commits on megarepo mainline.
This is not great since it breaks bisects, and this diff avoids that.
The way it does it is the following:
1) First do the same thing we did before - create deletion commit, and then
create a merge commit with p1 as deletion commit and p2 as an addition commit.
Let's call it "fake merge", since this commit won't be used for our mainline
2) Generate manifest for our "fake merge", and then use this manifest to
generate bonsai diff. But this time make p1 an old target commit (i.e. remove
deleted commit as if it never exited).
3) Use generated bonsai diff to create a commit.
So in short we split the procedure in two - first generate and validate the
resulting manifest (this is what we use "fake merge" commit for), and then
generate bonsai changeset using this manifest. It's unfortunate that in order
to generate resulting manifest we actually need to create commit and save it to
blobstore. If we had in-memory manifests we could have avoided that, but alas
we don't have them yet.
This way of creating bonsai changesets is a bit unconventional, but I think it has the benefit of relying on tools that we have confidence that they work (i.e. bonsai_diff), and we don't need to reimplement all the bonsai logic again
Reviewed By: mitrandir77
Differential Revision: D29633340
fbshipit-source-id: eebdb0e4db5abbab9346c575b662b7bb467497c4
Summary:
Initially I just wanted to address comments from D29515737 (fa8796ae19) about unnecessary
manifest retraversals, but there were a few more problems:
1) We didn't detect file conflicts in the final merge commit correctly. For
example, if additions_merge commit added a file "dir/1.txt", but there's
already file "dir" in target changeset that we won't detect this problem.
2) What's worse is that we might produce invalid bonsai merge changeset
ourselves. Say, if we delete "source_1/dir/file.txt", and then add file
"source_1/dir" in additions merge commit then resulting bonsai changeset should
have "source_1/dir" entry in the bonsai changeset.
This diff does the following:
1) Adds more tests to cover different corner cases - some of them were failing
before this diff.
2) Improves logic to verify file conflicts
3) Instead of trying to generate correct merge bonsai changeset it simplifies
the task and creates a separate deletion commit.
Note that creating a deletion commit on the mainline is something we want to
avoid to not break bisects. This will be addressed in the next diff.
Reviewed By: mitrandir77
Differential Revision: D29633341
fbshipit-source-id: 8f755d852212fbce8f9331049bf836c1d0a4ef42
