Summary:
This simplifies both client and server code to make bonsai changeset uploading be simpler for snapshots, as we only need a single commit, no mutations, etc.
This will make it easier to validate the bonsai changeset on the next diff.
It is fine to change both client and server as this code is not still in production, so we don't need to worry about rollout order.
Reviewed By: StanislavGlebik
Differential Revision: D30044542
fbshipit-source-id: d14bf58d671bc3bb5ff54b07c21f1781a043e0cf
Summary:
Using bonsai upload on commitcloud was useful to get things started, but it's not really necessary, and by removing we can simplify things a bit, since for snapshots it's a single commit.
This diff removes the client code. The server code is still used by snapshots, but will be simplified in a following diff.
Reviewed By: liubov-dmitrieva
Differential Revision: D30018157
fbshipit-source-id: 98bf0d30959de0c63b01bd11442f0bb3c538fcc7
Summary:
These fields had placeholder values and are now populated properly. I found out about `workingctx` which has most information I need about uncommitted changes that will create the snapshot.
TODO on next diffs:
- Remove code for uploading commit cloud stuff via bonsai, and simplify API.
- Verify created bonsai is valid before persisting it
- Store files and bonsai blobs in the created ephemeral bubble
- Modify bonsai changeset objects so they can contain snapshot information
Reviewed By: liubov-dmitrieva
Differential Revision: D30013238
fbshipit-source-id: c509d25c59d6396d922435cc3a97fc837d3fe51b
Summary: Using changes from D29995429, this returns the upload token of the changeset upload in the uploadsnapshot response.
Reviewed By: StanislavGlebik
Differential Revision: D30012368
fbshipit-source-id: 5ca54763153a474d1ce3c38ddeaa0efff071b09c
Summary:
Preupload LFS blobs for big file via LFS protocol rather than new EdenApi Uploads.
The `store.upload` function "skips" non lfs stuff but uploads lfs stuff remotely via LFS protocol.
The lfs implementation also uploads only what is missing, no difference from our implementation.
So, it seems it should be called right before our new Eden Api uploads.
Once it is called, our EdenApi uploads recognises the blobs that have been already uploaded and LFS blobs won't go via our new upload protocol.
Reviewed By: DurhamG
Differential Revision: D30012307
fbshipit-source-id: 39f09055ded71dc69fa0b12fe0ac74ee1a7d3659
Summary: Left over of D29989181 (ab8d843ebd).
Reviewed By: kulshrax
Differential Revision: D30052695
fbshipit-source-id: 3e126055c76f9d9e48566708abe1407bd2fecaa3
Summary:
Code was very duplicated between ContentId, Sha1, Sha256. And in this stack I plan to implement even more hashes, so I made this macro which makes it really easy to do so.
I took the opportunity to make the inner field not public, and only accessible via from/into.
Reviewed By: liubov-dmitrieva
Differential Revision: D29992279
fbshipit-source-id: c0b7225a3634071a1b1513119ec516d14bd8fd9e
Summary:
On `createremote`, create a changeset using the create_bonsai_changesets method created earlier.
For now, this changeset is create with a bunch of placeholders, but I added todo for all of those things and will tackle them over the next diffs, otherwise this would be a massive diff and hard to review.
Reviewed By: liubov-dmitrieva
Differential Revision: D29990295
fbshipit-source-id: 6b4c97887c0b4c017c586bf0ea06f12df9d07d23
Summary: This makes the `createremote` command also upload modified files to Mononoke, which will later be used to populate the snapshot.
Reviewed By: liubov-dmitrieva
Differential Revision: D29989181
fbshipit-source-id: 5a3b8d7133d6b27ea291ca01d14432a38d92f866
Summary:
This starts adding very basic behaviour to the createremote command.
- Added a `uploadsnapshot` method to the python/rust api. This will be used by the `createremote` command. It will create a bubble, and upload a snapshot to it. For now it just creates a bubble. The request/response objects are still subject to change.
- Added basic code to `createremote` that calls `uploadsnapshot`. It gets the modified files, but for now does nothing with them. I believe I'll have to read their content on rust code, as they are not in the "hg filestore" I believe, as they're not commited.
Reviewed By: StanislavGlebik
Differential Revision: D29987594
fbshipit-source-id: d1e332bb6d1baf9e90efdd2173474e8f3ebcc0e7
Summary: Add the `Content-Encoding` response header to `ResponseMeta`. This will be used to help support compressed EdenAPI responses.
Reviewed By: andll
Differential Revision: D30002079
fbshipit-source-id: 9bdf867c83d4994480cffa23990b6d1e887a645c
Summary:
The "Calculating" step corresponds to comparing manifests to determine what files have changed. Previously it was just a spinner with no concept of progress. I've tweaked it to display the current tree depth as we are planning. This gives the user something to derive progress from. Depth refers to the directory depth as the checkout plan breadth-first searches across the current tree and destination tree. The time required to diff the two trees is normally proprotional to the depth of the repo, so this is a decent indicator.
Note that there is no "total" set for the progress bar since the max or average depth of a repo doesn't seem readily available. We can add that in the future to provide a true progress bar.
Reviewed By: andll
Differential Revision: D29948994
fbshipit-source-id: bdb5d7a868345d66b9812c2e56159bbf66e6daff
Summary:
When uploading exactly the same file more than once, let's only upload each blob once.
This is already done on the function that also uploads filenodes, but not here.
Reviewed By: liubov-dmitrieva
Differential Revision: D29941483
fbshipit-source-id: ef8509223a11816c1b6f1e7f376d05b96f074340
Summary:
There were 3 places that use the same type of response:
```
Response {
index: usize,
token: UploadToken,
}
```
This diff merges all of them by using a single `UploadTokensResponse`. I'm still using aliases (`use as`) for all of them, if desired I can rename everywhere to use the actual type `UploadTokensReponse`.
Reviewed By: liubov-dmitrieva
Differential Revision: D29878626
fbshipit-source-id: 92af2d4c40eae42edd0a8594642ef0b816df4feb
Summary:
## High level goal
This stack aims to add a way to upload commits directly using the bonsai format via edenapi, instead of using the hg format and converting on server.
The reason this is necessary is that snapshots will be uploaded on bonsai format directly, as hg format doesn't support them. So this is a stepping stone to do that, first being implemented on commit cloud upload, as that code already uses eden api, and later will be used by the snapshotting commands.
## This diff
Modifies `uploadfileblobs_py` API method so that it returns the indexed UploadToken's. This will be used on the client code to get the content ids from the upload files, which are necessary to specify when storing the commit in Bonsai format. This is calculated in Rust when uploading the files, and we reuse the result so it doesn't need to be calculated again.
It reuses the `UploadHgFilenodeResponse` struct, which has the exact same format it needs. That is a bit ugly, and its refactored in D29878626.
Reviewed By: liubov-dmitrieva
Differential Revision: D29879617
fbshipit-source-id: 4e7bda1e1160da11c83f43002530fd1aba08d46d
Summary:
## High level goal
This stack aims to add a way to upload commits directly using the bonsai format via edenapi, instead of using the hg format and converting on server.
The reason this is necessary is that snapshots will be uploaded on bonsai format directly, as hg format doesn't support them. So this is a stepping stone to do that, first being implemented on commit cloud upload, as that code already uses eden api, and later will be used by the snapshotting commands.
## This diff
This diff adds the new endpoint to the API that can be acessed via python code, and in turn calls the api implemented on the last diff.
Reviewed By: liubov-dmitrieva
Differential Revision: D29849962
fbshipit-source-id: 5a2a674aef1edd3b0d95cb2b45b02ef9c20aca48
Summary:
## High level goal
This stack aims to add a way to upload commits directly using the bonsai format via edenapi, instead of using the hg format and converting on server.
The reason this is necessary is that snapshots will be uploaded on bonsai format directly, as hg format doesn't support them. So this is a stepping stone to do that, first being implemented on commit cloud upload, as that code already uses eden api, and later will be used by the snapshotting commands.
## This diff
Uploads file contents without uploading filenodes. Will be used for uploading a
commit in bonsair format instead of mercurial format via EdenAPI.
This is basically the same method as before D29549091 (d327996144).
Reviewed By: liubov-dmitrieva
Differential Revision: D29799484
fbshipit-source-id: 136c058ebcd814f39c5b903f5d8bfef7ff6005dc
Summary: streampager has been upgraded to 0.10. 0.9 no longer exists.
Reviewed By: andll
Differential Revision: D29794320
fbshipit-source-id: 415d4bc8649236a49dd41bb922a636096ab67be7
Summary:
Clean up usage of callback or progress in the new code.
There is no need to use a callback or progress. They were used for legacy progress support. The new progress bar automatically applies to every HTTP request.
Reviewed By: yancouto
Differential Revision: D29710912
fbshipit-source-id: 33889d89680c90e63f4520626a166d0b39b67afb
Summary:
We have a variety of enums in revisionstore that all serve the same purpose. With this change, I've consolidated them to a single type.
Not 100% sure if `StoreType` is the best name, in scmstore I named it `LocalStoreType` to be more clear - open to suggestions. I also decided to keep the variants named `Local` and `Shared` instead of adopting the `Local` and `Cache` terminology I used in scmstore - I'd rather not change that unless we decide to change the terminology everywhere to avoid confusion.
Reviewed By: kulshrax
Differential Revision: D29598025
fbshipit-source-id: 76d5a02230a8c1e5327cc5d90bbcae70049f251d
Summary:
Introduce a basic implementation of per-backend metrics collection for `FileStore`.
Currently, only indexedlog, lfs, and contentstore backends are instrumented, and only with basic metrics. Additional metrics (size, elapsed time, lfs pointer hits) and additional backends (aux, edenapi, lfs server) will be added in a later change.
Reviewed By: kulshrax
Differential Revision: D29552888
fbshipit-source-id: 54267f4de6f14db046cfae271b5c25a6bb494d7c
Summary:
Support exchange of mutation information during changesets uploads
Add new api for mutation information.
Add implementation for this new api.
Add client side support.
Reviewed By: markbt
Differential Revision: D29661255
fbshipit-source-id: 1d8cfa356599c215460aee49dd0c78b11af987b8
Summary:
Implement using of uploading changesets in `hg cloud upload` command.
This is the last part for `hg cloud upload` - uploading changesets via Edenapi
test
```
```
# machine #2
liubovd {emoji:1f352} ~/fbsource
[15] → hg pull -r 0b6075b4bda143d5212c1525323fb285d96a1afb
pulling from mononoke://mononoke.c2p.facebook.net/fbsource
connected to twshared27150.03.cln3.facebook.com session RaIPDgvF6l8rmXkA
abort: 0b6075b4bda143d5212c1525323fb285d96a1afb not found!
```
```
# machine #1
devvm1006.cln0 {emoji:1f440} ~/fbsource/fbcode/eden/scm
[6] → EDENSCM_LOG="edenapi::client=info" ./hg cloud upload
Jul 11 13:26:26.322 INFO edenapi::client: Requesting lookup for 1 item(s)
commitcloud: head '0b6075b4bda1' hasn't been uploaded yet
Jul 11 13:26:26.472 INFO edenapi::client: Requesting lookup for 6 item(s)
commitcloud: queue 1 commit for upload
Jul 11 13:26:26.648 INFO edenapi::client: Requesting lookup for 1 item(s)
commitcloud: queue 0 files for upload
Jul 11 13:26:26.698 INFO edenapi::client: Requesting lookup for 4 item(s)
commitcloud: queue 4 trees for upload
Jul 11 13:26:27.393 INFO edenapi::client: Requesting trees upload for 4 item(s)
commitcloud: uploaded 4 trees
commitcloud: uploading commit '0b6075b4bda143d5212c1525323fb285d96a1afb'...
Jul 11 13:26:28.426 INFO edenapi::client: Requesting changesets upload for 1 item(s)
commitcloud: uploaded 1 commit
```
```
# machine #2
liubovd {emoji:1f352} ~/fbsource
[16] → hg pull -r 0b6075b4bda143d5212c1525323fb285d96a1afb
pulling from mononoke://mononoke.c2p.facebook.net/fbsource
connected to twshared16001.08.cln2.facebook.com session QCpy1x9yrflRF6xF
searching for changes
adding commits
adding manifests
adding file changes
added 895 commits with 0 changes to 0 files
(running background incremental repack)
prefetching trees for 4 commits
liubovd {emoji:1f352} ~/fbsource
[17] → hg up 0b6075b4bda143d5212c1525323fb285d96a1afb
warning: watchman has recently started (pid 93231) - operation will be slower than usual
connected to twshared32054.08.cln2.facebook.com session Hw91G8kRYzt4c5BV
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
liubovd {emoji:1f352} ~/fbsource
[18] → hg diff -c .
connected to twshared0965.07.cln2.facebook.com session rrYSvRM6pnBYZ2Fn
diff --git a/fbcode/eden/scm/test b/fbcode/eden/scm/test
new file mode 100644
--- /dev/null
+++ b/fbcode/eden/scm/test
@@ -0,0 +1,1 @@
+test
```
Initial perf wins:
Having a large stack of 6 commits (total 24 files changed), tested *adding a single line to a file at the top commit*. We can see at least 2X win but it should be more because I have tested with a local instance of edenapi service that runs on my devserver.
```
╷
╷ @ 5582fc8ee 6 minutes ago liubovd
╷ │ test
╷ │
╷ o d55f9bb65 86 minutes ago liubovd D29644738
╷ │ [hg] edenapi: Implement using of uploading changesets in `hg cloud upload` command
╷ │
╷ o 561149783 Friday at 15:10 liubovd D29644797
╷ │ [hg] edenapi: Add request handler for uploading hg changesets
╷ │
╷ o c3dda964a Friday at 15:10 liubovd D29644800
╷ │ [edenapi_service] Add new /:repo/upload/changesets endpoint
╷ │
╷ o 28ce2fa0c Friday at 15:10 liubovd D29644799
╷ │ [hg] edenapi/edenapi_service: Add new API for uploading Hg Changesets
╷ │
╷ o 13325b361 Yesterday at 15:23 liubovd D29644798
╭─╯ [edenapi_service] Implement uploading of hg changesets
```
```
# adding new line to a file test in the test commit, and then run:
devvm1006.cln0 {emoji:1f440} ~/fbsource/fbcode/eden/scm
[8] → time hg cloud upload
commitcloud: head '4e4f947d73e6' hasn't been uploaded yet
commitcloud: queue 1 commit for upload
commitcloud: queue 0 files for upload
commitcloud: queue 4 trees for upload
commitcloud: uploaded 4 trees
commitcloud: uploading commit '4e4f947d73e676b63df7c90c4e707d38e6d0a93b'...
commitcloud: uploaded 1 commit
real 0m3.778s
user 0m0.017s
sys 0m0.027s
```
```
# adding another new line to a file test in the test commit, and then run:
devvm1006.cln0 {emoji:1f440} ~/fbsource/fbcode/eden/scm
[11] → time hg cloud backup
connected to twshared30574.02.cln2.facebook.com session uvOvhxtBfeM7pMgl
backing up stack rooted at 13325b3612d2
commitcloud: backed up 1 commit
real 0m7.507s
user 0m0.013s
sys 0m0.030s
```
Test force mode of the new command that reupload everything:
```
devvm1006.cln0 {emoji:1f440} ~/fbsource/fbcode/eden/scm
[13] → time hg cloud upload --force
commitcloud: head '5582fc8ee382' hasn't been uploaded yet
commitcloud: queue 6 commits for upload
commitcloud: queue 24 files for upload
commitcloud: uploaded 24 files
commitcloud: queue 61 trees for upload
commitcloud: uploaded 61 trees
commitcloud: uploading commit '13325b3612d20c176923d1aab8a28383cea2ba9a'...
commitcloud: uploading commit '28ce2fa0c6a02de57cdc732db742fd5c8f2611ad'...
commitcloud: uploading commit 'c3dda964a71b65f01fc4ccadc9429ee887ea982c'...
commitcloud: uploading commit '561149783e2fb5916378fe27757dcc2077049f8c'...
commitcloud: uploading commit 'd55f9bb65a0829b1731baa686cb8a6e0c5500cc2'...
commitcloud: uploading commit '5582fc8ee382c4c367a057db2a1781377bf55ba4'...
commitcloud: uploaded 6 commits
real 0m7.830s
user 0m0.011s
sys 0m0.032s
```
We can see the time is similar to the current `hg cloud backup` command.
Reviewed By: markbt
Differential Revision: D29644738
fbshipit-source-id: cbbfcb2e8018f83f323f447848b3b6045baf47c5
Summary:
Original commit changeset: 53d067bc2bb9
Same as https://www.internalfb.com/diff/D29586388 (0be0a68cca).
D29586388 (0be0a68cca) was disconnected from its parent and landed too early by mistake and reverted (because it wouldn't even compile without the parent commit).
This is just another version for the D29586388 (0be0a68cca) as it's not possible to reopen the original one.
Reviewed By: markbt
Differential Revision: D29598758
fbshipit-source-id: 96b6137366e196f9cd3c35c178eaa3d2fce1e071
Summary:
Add new client side API for upload trees.
Before uploading them, check what is already present on the server, similar as we check for filenodes.
I also added --force flag for the `hg cloud upload` command. It should be useful in general and useful for testing.
Reviewed By: markbt
Differential Revision: D29586388
fbshipit-source-id: 73c549f1a0d4328a64a133ab508fb4d253a4c33d
Summary:
upload filenodes (client side)
On the client side I implemented file upload and filenodes upload in the same API repo.edenapi.uploadfiles
This is because we should use the tokens from the file upload part to feed then into filenodes upload request.
Reviewed By: markbt
Differential Revision: D29549091
fbshipit-source-id: 436de187c8dce9a603c0c0a182e88b582a2d8001
Summary:
cpython-ext is not part of hg business APIs. It does not need to be consistent
with lower-case class name (and hg codebase is okay with non-lowercase class names,
just that most classes there are lower-case).
This resolves a rustc warning about the struct name.
Reviewed By: kulshrax
Differential Revision: D29526579
fbshipit-source-id: a4bc8e788d55c65aae9eaa87e2c684c2fded7ae2
Summary:
Since we pass edenapi to the contentstore, it should be doing the http
fetching internally. Let's get rid of the http fetching from the python logic
then.
Once we're 100% on edenapi, this will let us just delete the remotetreestore
entirely.
Reviewed By: quark-zju
Differential Revision: D29167443
fbshipit-source-id: fe1c4d7aae38135bdebf41a39c8beec34153e60e
Summary:
Eliminate remaining uses of `contentstore` directly in the Python bindings.
Add repack functionality to `LegacyStore`.
Implement repack methods on `FileStore` by calling into contentstore fallback if possible, otherwise directly.
Reviewed By: kulshrax
Differential Revision: D29469901
fbshipit-source-id: 5cd134d25680bb55b4ddbe1f6d69e02af8942b16
Summary: Will be used by the next change.
Reviewed By: andll
Differential Revision: D29434722
fbshipit-source-id: 74dbec506fb0985379480815380118cd41058aec
Summary: `scmstore` is dead, long live `scmstore`.
Reviewed By: kulshrax
Differential Revision: D29405613
fbshipit-source-id: 3252a545f5b944d14c15b2a777b84a99a2d4c293
Summary: This will be used by the next change.
Reviewed By: andll
Differential Revision: D29400533
fbshipit-source-id: e6b90bedd8d8a6cf9452dfb5c5f14f9980e12f62
Summary: This was used to narrow down issues.
Reviewed By: andll
Differential Revision: D29404054
fbshipit-source-id: 3bfdac332d63bdb13f40d5cf23dacec242b46d52
Summary:
The logic was wrong. If local is True, vertexes is a filtered list, while nodes
isn't. They cannot be mix-used.
Reviewed By: andll
Differential Revision: D29403466
fbshipit-source-id: 06f0853282b0f284d5b2970bfa37589ef59f6169
Summary:
Straightforward conversion of native checkout implementation from old scmstore API to new (non-async, batched) scmstore API.
We'll meet again someday soon, async.
Reviewed By: andll
Differential Revision: D29321512
fbshipit-source-id: 1e3e0d92c95730a5c2df610061f6faf5b1eb9068
Summary:
Make it possible for Python to detect the error and fallback to slow path
accordingly.
Reviewed By: andll
Differential Revision: D29369067
fbshipit-source-id: 1de6162532b145469c09070bdf843ec96d38e431
Summary: Previously, it was not possible to interrupt `hg` during EdenAPI fetch operations. This made it impossible to interrupt long-running fetches, which is very frustating to users. This can be simply fixed by using `block_unless_interrupted` in place of `block_on`.
Reviewed By: quark-zju
Differential Revision: D29344670
fbshipit-source-id: 3b0d36dda28f5f7cc812a07981f295f8d0fbdd8a
Summary:
Introduce basic contentstore fallback tracking to help monitor the scmstore shim rollout.
This will be expanded to a general fetch metrics system for scmstore in a future change.
Reviewed By: kulshrax
Differential Revision: D29305839
fbshipit-source-id: c6cc3ea15a3bb7b90f4ec298febc911ec4e2af91
Summary:
Introduce `LegacyStore` trait, which contains ContentStore methods not covered by other datastore traits.
Implement this trait for both contentstore and scmstore, and modify rust code which consumes `contentstore` directly to use `PyObject` and `LegacyStore` to abstract over both contentstore and scmstore instead.
Reviewed By: DurhamG
Differential Revision: D29043162
fbshipit-source-id: 26e10b23efc423265d47a8a13b25f223dbaef25c
Summary:
For a while we had two methods in async runtime: block_on_future and block_on_exclusive, due to historic reasons
Recently those methods were calling same code, and now it is time to replace both of them and rename to block_on
Reviewed By: quark-zju
Differential Revision: D29121107
fbshipit-source-id: 5faa76ae181e491b55d799c23c9de1b4e80298f3
Summary:
implement uploading file content via Eden API
* in this diff I aim to upload file content for the given set of filenodes
* also, the code would check with Mononoke using the lookup request what is already there and skip those
* also, this diff introduces calculation of blake2 hash (called ContentId) for file contents (we would probably need to store/cache those and the mapping from hg filenode id to the canonical Mononoke content_id)
* for every uploaded content EdenApi returns a token that we would also need to store later
Reviewed By: markbt
Differential Revision: D29063229
fbshipit-source-id: 739a44bc3ff904cb04a39514ba5efd01c80ba6d0
Summary: This will be used by upcoming changes.
Reviewed By: andll
Differential Revision: D29114048
fbshipit-source-id: ce963a42ebc5722ca9c61ab0a4e7a7377bbf7abb
Summary:
Modify the `ExtractInnerRef` implementations for `filescmstore` and `treescmstore` to return the new scmstore objects rather than the deprecated ones.
Modify `pycheckout` to use a new methods, `get_oldscmstore`, to access the deprecated scmstore instead (native checkout will be updated to use the new scmstore later on).
Reviewed By: DurhamG
Differential Revision: D29056498
fbshipit-source-id: 09b4655941bca048b01e6d365fcca1ccfb1cd700
