Summary: Log all thrift calls to scuba. Since the params are in thrift format we log them both in json and human readable format (e.g. path is binary thus not really readable in json format). In addition both http and thrift now log the request type.
Reviewed By: StanislavGlebik
Differential Revision: D13730245
fbshipit-source-id: 3419ec067a9066e181210d184195fbd02980c1e0
Summary: `BlobRepo` is cheaply clonable and doesn't need to be `Arc`'d.
Reviewed By: StanislavGlebik
Differential Revision: D13726727
fbshipit-source-id: 0b983c7c4625f47df8a11da5781b7777cc38d72f
Summary:
Reformat several rust files with the current rustfmt, to make the linter
happy.
Reviewed By: yfeldblum
Differential Revision: D13683205
fbshipit-source-id: f7a02dae0fbe095b6acde4de380aca2acfedf39d
Summary: MyRouter and cache support will come in next diffs
Reviewed By: StanislavGlebik
Differential Revision: D13465146
fbshipit-source-id: 0ede0e875d4a49794ff71173bd0d1563abb3ff08
Summary:
We need to add python exports to our TARGETS file to enable scmquery
to create a ServiceRouter Client for mononoke-api
Reviewed By: StanislavGlebik
Differential Revision: D13651275
fbshipit-source-id: ba5a4eb3665dae1ea127f4ceb96c1ce62e1e4563
Summary: There's nothing Mercurial-specific about identifying a repo. This also outright removes some dependencies on mercurial-types.
Reviewed By: StanislavGlebik
Differential Revision: D13512616
fbshipit-source-id: 4496a93a8d4e56cd6ca319dfd8effc71e694ff3e
Summary:
We've moved to the new config structure, I forgot to make the same changes to
apiserver
Reviewed By: lukaspiatkowski
Differential Revision: D13358407
fbshipit-source-id: cec81a21518cdb3c91dabb93e220d1ba3e25d02c
Summary:
`env_logger` was enabled in D12934682 to assist with debugging production TLS
issues with the Mononoke API server VIP. It turns out that this was actually
not neccesary due to the slog_stdlog crate. As such, remove env_logger from the
API server.
Reviewed By: StanislavGlebik
Differential Revision: D13328349
fbshipit-source-id: c5ed33d5afe96f4823ce637de892188e95c647fa
Summary: Restructure the configs so that we can specify more than one blobstore
Reviewed By: lukaspiatkowski
Differential Revision: D13234286
fbshipit-source-id: a98ede17921ed6148add570288ac23636b086398
Summary: Set the proper context for sessions.
Reviewed By: liubov-dmitrieva
Differential Revision: D13258641
fbshipit-source-id: edd18d4abc8f5475e0d2ac8395dfc877b2dd5958
Summary:
Config repo proved to be tricky to understand and hard to use. Let's just use
toml files.
Reviewed By: farnz
Differential Revision: D13179926
fbshipit-source-id: 3a44ee08c37284cc4c189c74b5c369ce82651cc6
Summary:
Let's add a command that builds and reads a skiplist indexes. This indexes will
be used by getbundle wireproto request to decrease the latency and cpu usage.
Note that we are saving only the longest "jump" from the skiplist. This is done
in order to save space.
Reviewed By: jsgf
Differential Revision: D13169018
fbshipit-source-id: 4d654284b0c0d8a579444816781419ba6ad86baa
Summary:
Let's make it use the same ChangesetFethcer as getbundle already does. It will
be used in the next diffs
Reviewed By: lukaspiatkowski
Differential Revision: D13122344
fbshipit-source-id: 37eba612935a209098a245f4be0af3bc18c5787e
Summary:
Most of our revsets are already migrated, let's migrate skiplists as well since
we want to use them in getbundle requests.
Reviewed By: lukaspiatkowski
Differential Revision: D13083910
fbshipit-source-id: 4c3bc40ccff95c3231c76b9e920af5db31b80d01
Summary: Now that Tupperware is using `/health_check`, there are no users of `/status`, so remove this endpoint.
Reviewed By: StanislavGlebik
Differential Revision: D12908785
fbshipit-source-id: be9bfae9453143a6f4b26f7fc6cbc68a3f1adc5c
Summary:
Mononoke-apiserver now will parse http header in order to get "host" url.
Based on that Host url, batch request will return the link to the same host.
Requirement:
LFS protocol consists of two parts:
- "batch" request, GET request
- HG client sends ids of the file it wants to download/upload
- API server should return valid url for downloading and uploading objects
"upload/download" request.
- Actual upload/download of the file
This diff allows API Server to return the link to itself on "batch" request. So Hg client will go to API Server again for uploading and downloading files.
Reviewed By: kulshrax
Differential Revision: D12941635
fbshipit-source-id: e56453b0d6239daa3848c285a1df09a4a869f2c8
Summary: We need more insight into what's happening inside of actix and tokio in order to debug operational issues with the API server. Unfortunately, this sort of low-level debug logs from libraries are not captured by slog. Since many of the API server's dependencies use `env_logger`, calling `env_logger::init()` in main will allow us to see the debug output from those libraries by specifying the `RUST_LOG` environment variable.
Reviewed By: HarveyHunt
Differential Revision: D12934682
fbshipit-source-id: 417201e509de66c3340c900a4914ea3a03a791fe
Summary:
This augments `/tree` to yields the size and content sha1 hash for the entries.
This is important for Eden and avoids additional round trips to the server.
The content hashing is the portion that I expect some push back on,
because it doesn't appear to be cached today and the implementation
here does a simplistic fetch and hash. By doing this we hope to
squash out a potential later fetch of the entire contents when
buck build is run.
Differential Revision: D10865588
fbshipit-source-id: c020ef07b99d8a5e8b2f8f7b699bf15e750d60a5
Summary:
troubleshooting startup problems is overly difficult without
printing more context, so print it.
Reviewed By: Anastasiya-Zhyrkevich
Differential Revision: D12814794
fbshipit-source-id: e815a6a93b4d1d3d03370b158f6fdc93edbc4ef5
Summary:
Background:
According to [git lfs protocol](https://github.com/git-lfs/git-lfs/blob/master/docs/api/batch.md) HTTP POST "batch" request should return a link to
the look-aside server.
In our case Mononoke API server is a look-aside server, and process both "batch" request and "upload/download" requests.
So it need to return a link to itself.
New approach requests a separate lfs-url for "batch" request.
The previous approach requested attributes --http-host and --http-port to make a link to the instance of API server running.
Reviewed By: StanislavGlebik
Differential Revision: D10488586
fbshipit-source-id: ed9d78ee9bc78bdcec5eea813bd9aaa6e4590a5c
Summary:
For unification with commit cloud vip configuration apiserver should support the same health check api
Request needed for corp2prod
The same as : D10488369
Reviewed By: liubov-dmitrieva
Differential Revision: D10488494
fbshipit-source-id: 50b4024295c596342a8080474383de850bb7754a
Summary:
As per the comments added - MyRouter setup is such that it starts inside a tupperware container together with the binary that will be using it. This means that by the time the binary wants to use the MyRouter connection the MyRouter instance might not be ready yet. In order to mitigate this effect the myrouter::Builder will attempt to make a "Select 1" query and retry it with a backoff for a max of 2 min or until the connection is actually established.
Unfortunately the `queries!` macro had to be moved inside the `macro` module in order to make it usable from inside `myrouter` module, see this: https://stackoverflow.com/questions/31103213/import-macro-from-parent-module
Reviewed By: farnz
Differential Revision: D10270464
fbshipit-source-id: 9cf6ad936a0cabd72967fb96796d4af3bab25822
Summary: Make get_manifest_by_nodeid accept HgManifestId and correct all calls to get_manifest_by_nodeid.
Reviewed By: StanislavGlebik
Differential Revision: D10298425
fbshipit-source-id: 932e2a896657575c8998e5151ae34a96c164e2b2
Summary:
The idea for rollout is to:
- first make sure that Mononoke doesn't crash when a --myrouter-port is provided
- then tupperware configs will be modified to include myrouter as a collocated proces on every host and the port of that myrouter instance will be provided via command line
- lastly land the change that actually talks to myrouter
Reviewed By: StanislavGlebik
Differential Revision: D10258251
fbshipit-source-id: ea9d461b401d41ef624304084014c2227968d33f
Summary:
Test is failing, as Mononoke server lfs support is not implemented yet.
Integration test for commands from hg client to Mononoke server.
\s(re) lines are added as after auto-save, the test script is formatted, and delete spaces at the empty lines.
In order to keep such lines, \s(re) could be added
In comparison of such line, pattern \s(re) is deleted and not compared.
See to mononoke/tests/integration/third_party/hg_run_tests.py for more information about comparison of the output lines.
Reviewed By: StanislavGlebik
Differential Revision: D10089289
fbshipit-source-id: 2962e80d919c21801d08990be190f2574c48646d
Summary:
PUT request upload to mononoke API
hg client sends a PUT request to store a file into blobstore during push supporting LFS
Upload file by alias is divied into 2 parts:
- Put alias : blobstore key
- Put blobstore_key: contents
Keep in mind, that file content is thrift encoded
host_address for batch request is from command line flags -H for host, -p for port
Reviewed By: StanislavGlebik
Differential Revision: D10026683
fbshipit-source-id: 6c2726c7fee2fb171582bdcf7ce86b22b0130660
Summary:
Previously cachelib cmdline args were added only to cmd line binaries, but not
to Mononoke this diff fixes it.
Reviewed By: farnz
Differential Revision: D10083899
fbshipit-source-id: 8febba96561c5ab9a61f60fafc7a7e56985dc038
Summary:
JSON blobs let other users of Mononoke learn what they need to know
about commits. When we get a commit, log a JSON blob to Scribe that other users can pick up to learn what they want to know.
Because Scribe does not guarantee ordering, and can sometimes lose messages, each message includes enough data to allow a tailer that wants to know about all commits to follow backwards and detect lost messages (and thus fix them up locally). It's expected that tailers will either sample this data, or have their own state that they can use to detect missing commits.
Reviewed By: StanislavGlebik
Differential Revision: D9995985
fbshipit-source-id: 527b6b8e1ea7f5268ce4ce4490738e085eeeac72
Summary:
POST request mononoke_api/objects/batch from hg client.
According to git-lfs protocol
https://github.com/git-lfs/git-lfs/tree/master/docs/apihttps://github.com/git-lfs/git-lfs/blob/master/docs/api/batch.md
In order to get url for uploading/downloading files, hg client is sending POST request mononoke_api/objects/batch of the following format.
This diff implements support for this POST request.
As an answer it returns json in the format required in git-lfs protocol (see link for more info).
Reviewed By: StanislavGlebik
Differential Revision: D9966691
fbshipit-source-id: 53bcbb4b455e61d9d344bfd9b5b6fb00bc201084
Summary:
WIP
Mononoke API download for lfs
support get request
curl http://127.0.0.1:8000/{repo_name}/lfs/download/{sha256}
Reviewed By: StanislavGlebik
Differential Revision: D9850413
fbshipit-source-id: 4d756679716893b2b9c8ee877433cd443df52285
Summary:
The "path" in manifold blobrepo is used for logging, but it has been quite confusing with "fbsource" and "fbsource-pushrebase" to be logged in an identical way - both are "fbsource", because of the "path" config. Lets not use the "path" for logging, instead use the "reponame" from metaconfig repo.
In case we ever want to have two repos that are named the same (please don't) or have logging under a different name than "reponame" from config then we can add a proper optional "name" parameter, but for now we don't require this confusing feature.
Reviewed By: StanislavGlebik
Differential Revision: D9769514
fbshipit-source-id: 89f2291df90a7396749e127d8985dc12e61f4af4
Summary:
Use the err_downcast macros instead of manual downcasting. Doesn't make
a huge code-size difference in this case, but a little neater?
Reviewed By: kulshrax, fanzeyi
Differential Revision: D9405014
fbshipit-source-id: 170665f3ec3e78819c5c8a78d458636de253bb6f
Summary: While I was working on `actix-srserver`, I realized the current design of the API server is quite unnecessary. The "MononokeActor" and "MononokeRepoActor" are only returning futures without much CPU computation cost. So it don't need to be placed in a separate thread.
Reviewed By: jsgf
Differential Revision: D9472848
fbshipit-source-id: 618ec39c42d90717fa6985fee7d6308420962d3f
Summary: Added a thrift client library and binary for Mononoke API Server that allows us to play with the API Server's thrift port.
Reviewed By: farnz
Differential Revision: D9110899
fbshipit-source-id: 603cc5e2b5e0419a73c9eccb35f8c95455ada9ce
Summary: This commit adds a basic thrift server that responds to fb303 status check queries to Mononoke API Server.
Reviewed By: farnz
Differential Revision: D9092291
fbshipit-source-id: d1e4ddb280c252f549d40a0bb03d05afccbf73b8
Summary: Adds proper url decoding for is_ancestor, so that special characters can be encoded in the url.
Reviewed By: kulshrax
Differential Revision: D9325467
fbshipit-source-id: d3ff60e004be8d254ea6f7288188adf54ab7ff5f
Summary:
We'll be running in Tupperware, and want to shrink when we get too
large to avoid OOM due to caches. Configure cachelib appropriately
Reviewed By: StanislavGlebik
Differential Revision: D8900371
fbshipit-source-id: 4f1f64c2508c64e4ce2d201e0a0e86446f84ffef
Summary: This fixes the `blocking` cannot run without threadpool error.
Reviewed By: farnz
Differential Revision: D9017757
fbshipit-source-id: 037fd6f30598f56a83c1dd91c9b8c4f3c8e413b3
