Summary: The `/gethgfiles` and `/getfilehistory` endpoints were only used by the Eden API Hyper client, which no longer exists. As such, let's clean up these old endpoints to avoid cluttering the API server code.
Reviewed By: fanzeyi
Differential Revision: D15508314
fbshipit-source-id: f4f76baf5645750a6da9b28897052151f6183e60
Summary: Add an endpoint to the API server that provides functionality similar to the `gettreepack` wire protocol command.
Reviewed By: fanzeyi
Differential Revision: D15492734
fbshipit-source-id: 7d0f113f0d33c68d5bfba5781269a92f0d5a66e8
Summary: Change the API so that the act of getting the data from a data entry (optionally) forces hash validation.
Reviewed By: xavierd
Differential Revision: D15405742
fbshipit-source-id: c127286322c707eb066f89021ac45315e89aa718
Summary:
As part of adding support for infinitepush in Mononoke, we'll include additional server-side metadata on Bookmarks (specifically, whether they are publishing and pull-default).
However, we do use the name `Bookmark` right now to just reference a Bookmark name. This patch updates all reference to `Bookmark` to `BookmarkName` in order to free up `Bookmark`.
Reviewed By: StanislavGlebik
Differential Revision: D15364674
fbshipit-source-id: 126142e24e4361c19d1a6e20daa28bc793fb8686
Summary: Add a new `/repo/eden/trees` endpoint to the API server that serves tree data in Mercurial's tree format. This endpoint uses the same structs used by the file data endpoint for framing the returned tree data.
Reviewed By: xavierd
Differential Revision: D15288534
fbshipit-source-id: 9d31b77d3714ba815b4f3b86fa75b0c5ab1a7469
Summary: Given that the on-the-wire representation of file and tree data is essentially the same, we can reuse our existing structs for file fetching for tree fetching. As such, we should rename these structs to make it clear that they are not just for file data. (This also involves updating some comments/variable names to make sure everything is consistent.)
Reviewed By: singhsrb
Differential Revision: D15286690
fbshipit-source-id: 8c7fa4392057e90a9f19beb17b0bbcbf04b7e8e7
Summary: As we add more functionality to the Eden API, we will have a lot more request structs. These structs are only used by the HTTP data fetching code, and should not be used by actual business logic. As such, while these types need to be public (so that both Mononoke and Mercurial can use them), they should not be re-exported at the top level.
Reviewed By: quark-zju
Differential Revision: D15268439
fbshipit-source-id: e7d1405d2ac234892baedbf7dbf3e133d187cb45
Summary:
This migrates the internal structures representing the repo and storage config,
while retaining the existing config file format.
The `RepoType` type has been replaced by `BlobConfig`, an enum containing all
the config information for all the supported blobstores. In addition there's
the `StorageConfig` type which includes `BlobConfig`, and also
`MetadataDBConfig` for the local or remote SQL database for metadata.
Reviewed By: StanislavGlebik
Differential Revision: D15065421
fbshipit-source-id: 47636074fceb6a7e35524f667376a5bb05bd8612
Summary:
Given that users typically connect to the API server via a VIP, it is
often unclear which actual host we're connected to. As such, API server
responses should contain the actual hostname of the server.
As a first pass at adding this, let's add a simple endpoint that just returns
the hostname, which can be used to see which hosts the VIP is sending requests
to.
Reviewed By: xavierd
Differential Revision: D15168952
fbshipit-source-id: d6fee5716df975fa1588bbf069ad6e970a779b1e
Summary: The SHA1 cache mistakenly caching only by the node hash while it is possible to have multiple files that have the same node hash while different names. This will cause the server incorrectly returning incorrect entry from the cache.
Reviewed By: StanislavGlebik
Differential Revision: D15092963
fbshipit-source-id: 608be9f2b749d0b77687ad5cf2d76d56398a543d
Summary: Now it is possible to configure and enable/disable bookmark cache from configs
Reviewed By: StanislavGlebik
Differential Revision: D14952840
fbshipit-source-id: 3080f7ca4639da00d413a949547705ad480772f7
Summary: Update the Eden API client and server to use the new DataEntry type for file content downloads.
Reviewed By: quark-zju
Differential Revision: D14907958
fbshipit-source-id: 8a7b1cbb54bdc119dda11179ff94d3efdb7e85c9
Summary: Removing this function in favor of using Key.path
Reviewed By: quark-zju
Differential Revision: D14945331
fbshipit-source-id: 6b6bb70375629edf37b2b04a86545f18e15b33b4
Summary:
Previously, the Eden API endpoints on the API server used JSON for encoding requests/responses for debugging purposes. Given that these APIs are mostly used to transfer large amounts of binary data, we should use a binary encoding scheme in production. CBOR fits the bill since it is essentially binary JSON, allowing for more efficient coding of large byte strings.
Although this is a breaking API change, given that nothing depends on these endpoints yet, it should be OK to simply change the format.
Reviewed By: xavierd
Differential Revision: D14907978
fbshipit-source-id: e0aea30d7304f4b727e2ad7fe23379457b6c3e26
Summary:
This function is difficult to justify in the context of the Rust borrow checker.
The primary concern for this pattern is preventing mutation when the object is
passed around.
We can always add the function back if it has to more than just return the
underlying value.
Reviewed By: quark-zju
Differential Revision: D14877545
fbshipit-source-id: acdd796e1bee5445c1bce5ce0ceb41a7334e4966
Summary: Add new API server endpoints for performing batch fetches of data and history. The requests and responses are currently serialized as JSON, though the format can be trivially changed later to something like CBOR (or any other format supported by serde).
Reviewed By: xavierd
Differential Revision: D14573735
fbshipit-source-id: ac46aa8a55db44b695c53f9e32cc4ea2e17504c8
Summary:
getpackv1 needs to return copy metadata together with the content. This diff
fixes it
Reviewed By: kulshrax
Differential Revision: D14668319
fbshipit-source-id: 56a6ea2eb3a116433446773fa4fbe1f5a66c5746
Summary: Implement a backing store that uses thrift protocol to communicate with Mononoke API Server.
Reviewed By: strager
Differential Revision: D13966575
fbshipit-source-id: 66f66dda6b17aecd6c6b4475ab6b004c608f457f
Summary: Add some logging to the API server during repo intiialization, since startup of the API server in non-opt modes takes a long time.
Reviewed By: quark-zju
Differential Revision: D14605643
fbshipit-source-id: 4b3c46f16cbeb3bf571e553eadc4cb002ce044d9
Summary:
Is_ancestor was inexplicably slow so I added a few perf counters to it. The slowness should be fixed with D14540318.
This will only add the most important perf counters as I have seen some slow down when adding more.
I also changed the param naming from maybe_ancestor to ancestor. The maybe is implied through the method name being is_ancestor. This change also brings skiplist in line with apiserver
What will we log:
Ancestor/Descendant Generation
skip/noskip iterations
skipped generations
Reviewed By: StanislavGlebik
Differential Revision: D14085659
fbshipit-source-id: e0c7d34ed218e94c2c5a39f98d3c0b3d4265214f
Summary: All of the API server handlers were unnecessarily cloning URL parameters. This diff eliminates these clones.
Reviewed By: singhsrb
Differential Revision: D14537692
fbshipit-source-id: 02ea9dccae02b2813d04dd95feb8225e35870b6c
Summary: `LooseHistoryEntry` and `PackHistoryEntry` aren't the best names for these types, since the latter is what most users should use, whereas the former should only typically used for data transmission. As such, we should rename these to clarify the intent.
Differential Revision: D14512749
fbshipit-source-id: 5293df89766825077b2ba07224297b958bf46002
Summary: See D14279065, this diff is simply to clean up the deprecated code
Reviewed By: StanislavGlebik
Differential Revision: D14279210
fbshipit-source-id: 10801fb04ad533a80bb7a2f9dcdf3ee5906aa68d
Summary: The username and ssh vars are both properties of request, so they fit nicely inside per request CoreContext. They will be used in verify_integrity hook.
Reviewed By: StanislavGlebik
Differential Revision: D14385840
fbshipit-source-id: 9fe3cb96ffa89d8b017c730e37ca9ea4124ede0c
Summary: Add an optional `depth` parameter to the `getfilehistory` endpoint, allowing the caller to specify the maximum number of history entries to fetch.
Differential Revision: D14222991
fbshipit-source-id: 8b28c9674e764f0098a9780f94ef2957ef117f09
Summary: Optionally specify a max depth for history fetching.
Reviewed By: StanislavGlebik
Differential Revision: D14218337
fbshipit-source-id: b6b92181172637e58a43bf61793257559915c7f1
Summary:
Move most of the fb303 methods from `fb303.FacebookService` to
`fb303_core.BaseService`.
A few methods have been left behind. In most cases these are methods that are
either already deprecated or that we would like to deprecate in the future.
One important method that is not being moved as part of this diff is
`getStatus()`. We do want to move this to `BaseService` in the future, but
doing so is a bit more challenging than the other methods, since it also uses
a custom enum that will need to be moved from `fb303.thrift` to
`fb303_core.thrift`. In most languages this can be done relatively easily
by replacing it with a typedef in `fb303.thrift`. Unfortunately the `py3`
thrift generator does not handle typedefs well, so more work will need to be
done to enable this for python3 thrift code.
Reviewed By: yfeldblum
Differential Revision: D13688965
fbshipit-source-id: 48d3d007ef6b63b4176cab753624813b7f9d6526
Summary: This diff adds a new `getfilehistory` endpoint to the API server that takes a filenode/path pair and returns a stream of JSON-encoded history entries (corresponding to `types::LooseHistoryEntry`) representing the full history of that filenode. (Note that the particular serialization format can be changed relatively easily in the future; for now I've chosen JSON for ease of debugging.)
Reviewed By: StanislavGlebik
Differential Revision: D14197233
fbshipit-source-id: 390f2e528f36fa3ec4b402e2fc3def0f2d529432
Summary: Convert the API Server to Rust 2018 edition.
Reviewed By: StanislavGlebik
Differential Revision: D14222621
fbshipit-source-id: 4afd8a57c415cfd5b5923d15cb2873a1c098f861
Summary: The Cargo.toml file for the API server appears to be from a very old prototype version. It is tiny and barely captures any of the current dependencies (many of which can't be built with Cargo anyway). Let's delete it since it's not adding any value.
Reviewed By: StanislavGlebik
Differential Revision: D14222619
fbshipit-source-id: c7ce625da859ae6fa85e787a6e92a2795c8420c5
Summary:
The main module of the API server is growing pretty large, since a new method and query parameters struct is required for each new HTTP endpoint added to the server. To make this manageable, let's restructure the code so that each handler function is paired with a params struct that contains its corresponding params.
This makes it easier to make changes to individual handlers, such as switching the method GET to a POST and changing the handler to read the params from the request body.
Reviewed By: StanislavGlebik
Differential Revision: D14222620
fbshipit-source-id: 67abea2ff32a3d6cc278375efd368fa2969ab891
Summary: This diff added "manifest" field to Mononoke API Server's `get_changeset` method, which is required to implement Eden's `BackingStore::getTreeForCommit`
Reviewed By: StanislavGlebik
Differential Revision: D13966576
fbshipit-source-id: d2444b3aab477ab76efa7baddf114d90fa7a7a77
Summary: Add a thrift method `get_tree` to Mononoke API Server
Reviewed By: StanislavGlebik
Differential Revision: D13964085
fbshipit-source-id: c0d31e4e13079bec49d480c116ebb300c7e4260f
Summary: This is needed to enable D14085659. Currently we do not have access to the core context when actually logging stuff. This change will not affect http api as Im not sure how to send the context to the middleware.
Reviewed By: StanislavGlebik
Differential Revision: D14081760
fbshipit-source-id: 6c3e8e1e7d153fbac62c842130e9caf51e5aac5a
Summary: use common funcions to parse --myrouter-port, this is a simple clean up
Reviewed By: StanislavGlebik
Differential Revision: D14084003
fbshipit-source-id: 63d6c8301e977faead62cb1c705bac372d56594e
Summary: New commits should be logged to scribe, these will be used to trigger the update for the hg clone streamfile.
Reviewed By: lukaspiatkowski
Differential Revision: D14022599
fbshipit-source-id: a8a68f12a8dc1e65663d1ccf1a5eafa54ca2daf0
Summary:
Update the `fb303.FacebookService` type to derive from the new
`fb303_core.BaseService` definition. This is in preparation to begin moving
most of the fb303 methods to `BaseService`.
Unfortunately the Rust thrift compiler does not handle service inheritance
very well, so all existing rust thrift servers need to be updated to handle
this change.
Reviewed By: wez
Differential Revision: D13683717
fbshipit-source-id: c6ba8dcbadd05d674b7a75ea531500268e4bc3f2
Summary:
Populating skiplist at startup takes up to 3 minutes when not compiling in prod mode. This allows developers to turn this off and just use an empty skiplist (this only affects historical queries, and those can still be tested using recent commits)
The reason for negating the flag is to not hide core functionality behind a flag.
Reviewed By: StanislavGlebik
Differential Revision: D13986936
fbshipit-source-id: 053ecbd27aeeaa97f03194933e1d904a3e60ffcc
Summary:
I decided against using the naming convention maybe_ancestor and maybe_descendant as I believe the name is_ancestor already implies this uncertainty.
Skiplist is used instead of bfs, this improves performance significantly when using very old commits. For now we do not update skiplist as more recent commits are in cache anyway.
Reviewed By: StanislavGlebik
Differential Revision: D13917333
fbshipit-source-id: 21b49d920ff473c953a952ee3c6d7b55565f98ac
Summary: HgFileNodeId is a stronger typed id, so it is prefered to use it instead of HgNodeHash whenever it is identifying a filenode
Reviewed By: aslpavel
Differential Revision: D13986172
fbshipit-source-id: c0334652345acb868e86c38b8c0045e9c023c176
Summary: The Copy trait means that something is so cheap to copy that you don't even need to explicitly do `.clone()` on it. As it doesn't make much sense to pass &i64 it also doesn't make much sense to pass &<Something that is Copy>, so I have removed all the occurences of passing one of ouf hashes that are Copy.
Reviewed By: fanzeyi
Differential Revision: D13974622
fbshipit-source-id: 89efc1c1e29269cc2e77dcb124964265c344f519
Summary:
Apiserver should log success and not rely on scmquery-proxy for that. Also did some refactoring:
1. cleaned up .timed methods in the thrift section
2. Moved the creation of ScubaSampleBuilder to one central location so common logging data doesnt have to be added to both http and thrift section
Reviewed By: StanislavGlebik
Differential Revision: D13895685
fbshipit-source-id: 3c6295e172f8ccaabaf38f436b304b5cb717deca
