Summary:
getfiles implementation for lfs
The implementation is the following:
- get file size from file envelope (retrieve from manifold by HgNodeId)
- if file size > threshold from lfs config
- fetch file to memory, get sha256 of the file, will be fixed later, as this approach consumes a lot of memory, but we don't have any mapping from sha256 - blake2 [T35239107](https://our.intern.facebook.com/intern/tasks/?t=35239107)
- generate lfs metadata file according to [LfsPlan](https://www.mercurial-scm.org/wiki/LfsPlan)
- set metakeyflag (REVID_STORED_EXT) in the file header
- if file size < threshold, process usual way
Reviewed By: StanislavGlebik
Differential Revision: D10335988
fbshipit-source-id: 6a1ba671bae46159bcc16613f99a0e21cf3b5e3a
Summary: Make get_manifest_by_nodeid accept HgManifestId and correct all calls to get_manifest_by_nodeid.
Reviewed By: StanislavGlebik
Differential Revision: D10298425
fbshipit-source-id: 932e2a896657575c8998e5151ae34a96c164e2b2
Summary:
We now have a way for a MySQL database to tell us how to send
streaming clones to the client. Hook it all up, so that (with any luck), once
we have data in MySQL and the blobstore, we'll see working streaming clones.
Reviewed By: StanislavGlebik
Differential Revision: D10130774
fbshipit-source-id: b22ffb642d0a54b09545889779f79e7a0f81acd7
Summary:
JSON blobs let other users of Mononoke learn what they need to know
about commits. When we get a commit, log a JSON blob to Scribe that other users can pick up to learn what they want to know.
Because Scribe does not guarantee ordering, and can sometimes lose messages, each message includes enough data to allow a tailer that wants to know about all commits to follow backwards and detect lost messages (and thus fix them up locally). It's expected that tailers will either sample this data, or have their own state that they can use to detect missing commits.
Reviewed By: StanislavGlebik
Differential Revision: D9995985
fbshipit-source-id: 527b6b8e1ea7f5268ce4ce4490738e085eeeac72
Summary:
Streaming clones are a neat hack; we get to send files down to the
Mercurial client, which it then writes out as-is. Usefully, we can send no
files, and the protocol still works.
Set up the capabilities etc needed so that we send streaming clones to
Mercurial clients, even if they're rather useless so far.
Reviewed By: StanislavGlebik
Differential Revision: D9926967
fbshipit-source-id: b543e802adac38c8bc318081eec364aed87b0231
Summary: Should reduce memory usage and make getbundle a bit faster
Reviewed By: farnz
Differential Revision: D9861578
fbshipit-source-id: 57bca3700e3a38aeb70f267e6dc90d6b8a9d2955
Summary: Use `ChangesetId` in `DifferenceOfUnionsOfAncestorsNodeStream` instead of `HgNodeHash`. This avoids several bonsai lookups of parent nodes.
Reviewed By: StanislavGlebik
Differential Revision: D9631341
fbshipit-source-id: 1d1be7857bf4e84f9bf5ded70c28ede9fd3a2663
Summary:
The "path" in manifold blobrepo is used for logging, but it has been quite confusing with "fbsource" and "fbsource-pushrebase" to be logged in an identical way - both are "fbsource", because of the "path" config. Lets not use the "path" for logging, instead use the "reponame" from metaconfig repo.
In case we ever want to have two repos that are named the same (please don't) or have logging under a different name than "reponame" from config then we can add a proper optional "name" parameter, but for now we don't require this confusing feature.
Reviewed By: StanislavGlebik
Differential Revision: D9769514
fbshipit-source-id: 89f2291df90a7396749e127d8985dc12e61f4af4
Summary: We are going to use it in pushrebase as well
Reviewed By: lukaspiatkowski
Differential Revision: D9635432
fbshipit-source-id: 5cbe0879d002d9b6c21431b0938562357347a67f
Summary: In case `max_depth=1` we should only return the topmost entry, which in this case always is the root-entry. This fixes it so that we always return-fast in case `max_depth=1`.
Reviewed By: StanislavGlebik
Differential Revision: D9614259
fbshipit-source-id: a6b82bd5aac74d004f61a07bc24f5d26e5c56412
Summary:
Looks like we shouldn't have raised in the first place. Big getfiles buffer
causes OOMs on the servers. Also memory profiling shows that quite often most
of the Mononoke server's memory is used for serving remotefilelog requests.
Reviewed By: purplefox
Differential Revision: D9601990
fbshipit-source-id: 356a65d0749b064486436fb737bd5a47b3beecfa
Summary:
Modify the lookup() RPC function to be able to accept either a
bookmark or commit hash. A commit hash lookup is attempted first, falling
back to a bookmark lookup if it fails.
Reviewed By: StanislavGlebik
Differential Revision: D9457349
fbshipit-source-id: 78db21c01c498b045f5781097cb12f7220a40999
Summary: This commits change `HgBlob` from an enum into a struct that only contains one Bytes field, completely removes `HgBlobHash` and changes the methods of `HgBlob` from returning `Option`s into directly returning results.
Reviewed By: farnz
Differential Revision: D9317851
fbshipit-source-id: 48030a621874d628602b1c5d3327e635d721facf
Summary: Since this data is specific to TimedStream and not TimedFuture I split the Stats struct into FutureStats and StreamStats
Reviewed By: StanislavGlebik
Differential Revision: D9355421
fbshipit-source-id: cc2055706574756e2e53f3ccc57abfc50c3a02ba
Summary: gettreepack doesn't care for deleted entries, only about added or modified ones
Reviewed By: StanislavGlebik
Differential Revision: D9378909
fbshipit-source-id: 2935e6b74fbb0208f7cf89ab4b1e761bb9c6000b
Summary:
First step of implementing pushrebase algorithm. Save the commits that client
has sent us. The parts that client sends us are the same as in normal push
except for the names and parameters.
Reviewed By: farnz
Differential Revision: D9304750
fbshipit-source-id: d5be6635c0cf1a14a66a5fed5ba13f344195e8bc
Summary:
Caught by inspection - this builds up a long linked list of streams
to evaluate, where `select_all` has a vector and uses `FuturesUnordered` to
optimize which stream to check next.
Reviewed By: lukaspiatkowski
Differential Revision: D9359932
fbshipit-source-id: 83e645cc5f17a6376d0f0925b88cb390caf3dc4f
Summary: This is more general, and allows one to call `RepoClient` methods.
Reviewed By: farnz
Differential Revision: D9318658
fbshipit-source-id: 09b2e64bc0d423eafcb381902e03f349fc666a41
Summary: Remove `Arc<BlobRepo>` from more places since `BlobRepo` will do its own internal `Arc`ing.
Reviewed By: StanislavGlebik
Differential Revision: D9317987
fbshipit-source-id: 899e8b2ede278e62a83e64c144eb18c8cc7e57c6
Summary: The "path" here is only used for logging. Also add another constructor which can be used to provide custom naming.
Reviewed By: StanislavGlebik
Differential Revision: D9309237
fbshipit-source-id: 0192a3f74129ad063aa41ed61125b3b1d2711a77
Summary: all that places logged all list of HgNodeHash
Reviewed By: StanislavGlebik
Differential Revision: D9242904
fbshipit-source-id: 42a98b04986f2ed432a8956828356bd5b9bcaa88
Summary: the current 100 is just too small, which can be seen while tracing the requests
Reviewed By: farnz
Differential Revision: D9032184
fbshipit-source-id: adbd274d2bd275cc57af635ff42321f06fa358de
Summary: This makes it easier to look at traces as a whole client session, it also makes it possible to see how long did the entire request took time (previously it would not count the time to send data back).
Reviewed By: farnz
Differential Revision: D9032172
fbshipit-source-id: 048fcb21d4eb70424643f7bb2afa426e09904ada
Summary: As a bonus this diff also contains unifying the linknode family of methods (they all now accept arguments via reference) and better tracing for get_files request
Reviewed By: StanislavGlebik
Differential Revision: D9031283
fbshipit-source-id: 4526a8446984904bce7d4dcef240088c7f2ffaa3
Summary: Canaries are better used with ODS counters, so the important metrics (like latency) need to be logged to both scuba and ODS
Reviewed By: jsgf
Differential Revision: D9007402
fbshipit-source-id: 43c057dfdccc5639051d2adee90704de71fc2df7
Summary:
These types are Hg specific. Since we are going to add bonsai changeset
creation soon, let's make it clear in the types
Reviewed By: farnz
Differential Revision: D8911359
fbshipit-source-id: 8b6cc45122402d7b7e074e66d904d979030de705
Summary: Iterating over the code on server is a bit painful and it has grown a lot, splitting it should speed up future refactories and make it more maintainable
Reviewed By: jsgf, StanislavGlebik
Differential Revision: D8859811
fbshipit-source-id: 7c56f9f835f45eca322955cb3b9eadd87fbb30a1
