Summary: We are fetching envelopes twice in repo_commit, this updates the code to provide a shared abstraction over file and manifest envelopes in order to avoid this duplicate fetch.
Reviewed By: farnz
Differential Revision: D16282794
fbshipit-source-id: 53d53f28f051cbacbc1c931b2dbb98ac74a19c15
Summary:
In repo_commit, we already leverage a file or manifest's envelope to gets its parents, but were getting the copy info from the raw filenode bytes (through a call to `get_raw_content`, which for a filenode eventually calls into `fetch_raw_filenode_bytes` to get the content from the blobstore and retrieves the metadata from the envelope, and then proceeds to discard the contents and only retain the copy info).
This updates this code to just get the metadata from the envelope in order to retrieve copy info, which is functionally equivalent, but avoids a fetch to the blobstore.
There's still some potential for improvement here, since we're duplicating the calls to get the envelope for a file. I'm fixing this up in a separate diff for clarity.
Reviewed By: farnz
Differential Revision: D16282795
fbshipit-source-id: eb9068dcc0d36c4a9c0046e856db3fb2e9a3748f
Summary:
This diff is loosely based of D15851686.
There were a few problems with the previous load shedding api. The biggest
problem was that it used static variables that required explicit
initialization. So if someone forgets to add an init in any tool (say, in
mononoke_admin), and then calls the code that runs `should_throttle` then the
binary would just crash.
The second big problem was that on each `bump_load` call configerator config
was parsing json config over and over again. I didn't measure how expensive it
was, but it looked very counter-intuitive that we put load counters in static
variables (IIRC to avoid allocations), but then we reparse the config on each
call.
To avoid these problems this diff moves ratelimiting logic into CoreContext.
The config is parsed only once on CoreContext creation and all global variables
are now local. Because of these changes `loadlimiter` library is no longer
necessary.
Note that with load limiter counters are allocated on each
`bumpLoad/shouldThrottle` call. I'll investigate how expensive it actually is.
One drawback is that now it's trickier to track egressbytes for stderr, and it was removed.
We can add it later if needed
Reviewed By: farnz
Differential Revision: D16221607
fbshipit-source-id: f934cb2fc778df1679acc7d6eb2a1a8b4deb0aac
Summary: Implements `inspect_err` helper that allows to react to failed future without consuming it.
Reviewed By: StanislavGlebik
Differential Revision: D16225137
fbshipit-source-id: fe4678f029615e8ae42b30c6c908a31dfc5e8a86
Summary:
The API Server uses Mononoke's blobrepo code and therefore the
stats that they implement. However, without starting a task to aggregate the stats,
they will never be collected.
Spawn the required future on the tokio runtime to allow stats to be exported.
Reviewed By: farnz
Differential Revision: D16280673
fbshipit-source-id: b97efe22208b20b175e67da2c5c0c29f407d0e06
Summary:
This is the first step in adding unodes into Mononoke.
Unode is a datastructure that allows finding file/directory history easily.
Among other things it will be used by fastlog and blame.
This diff introduces core unode datastructures.
Reviewed By: krallin
Differential Revision: D16262587
fbshipit-source-id: 484800ea8d1ccb82ee9213adc5ba9de68fbaa9b2
Summary:
Instantiating a new DB connection may require remote calls to be made to e.g. Hipster to allocate a new certificate (this is only the case when connecting to MySQL).
Currently, our bindings to our underlying DB locator make a blocking call to pretend that this operaiton is synchronous: https://fburl.com/ytmljxkb
This isn't ideal, because this call might actually take time, and we might also occasionally want to retry it (we've had issues in our MySQL tests with acquiring certificates that retrying should resolve). Running this synchronously makes doing so inefficient.
This patch doesn't update that, but it fixes everything on the Rust side of things to stop expecting connections to return a `Result` (and to start expecting a Future instead).
In a follow up diff, I'll work on making the changes in common/rust/sql to start returning a Future here.
Reviewed By: StanislavGlebik
Differential Revision: D16221857
fbshipit-source-id: 263f9237ff9394477c65e455de91b19a9de24a20
Summary:
When the actual database schema was changed, this was left behind. Let's fix
this.
Reviewed By: StanislavGlebik
Differential Revision: D16254128
fbshipit-source-id: 9a5a0d453f8cc846e9f5f963aadb3a9e9b705ef8
Summary:
This patch adds optimized implementations of is_present() for FileBlob (which
we use in tests), and ManifoldBlob (which AFAIK, we don't use: I believe we use
ManifoldThriftBlob instead).
The underlying goal is to make sure we don't perform accidental reads of LFS
blobs when all we meant was to check a blob existed when committing a
changeset.
For ManifoldBlob, this is implemented by querying for an empty range, which
appears to be explicitly supported as per the Rust client's source code (it
sets `?withPayload=0` as a query parameter).
Reviewed By: StanislavGlebik
Differential Revision: D16262273
fbshipit-source-id: c055f63d114d40be2c448b8c8e34eb82d0e641b5
Summary:
This updates `find_file_changes` in bonsai generation to avoid content fetches. Until now, we were fetching content to see how big it is and compute its content ID, and sometimes we were also fetching the envelope to access copy info.
This updates the logic to instead unconditionally fetch the envelope, and not fetch the content at all.
Reviewed By: farnz
Differential Revision: D16262271
fbshipit-source-id: 5b299fe2769d3b8b584da2a7551434b7872396b2
Summary: This helps us investigate the workings of the (upcoming) xdb blobstore.
Reviewed By: StanislavGlebik
Differential Revision: D16208770
fbshipit-source-id: 33542f3d34a5c8b4287bb14b0aa97d3802b0e0d6
Summary:
Before this diff, the `RepoBlobstore` type alias (and the newly-added
`RepoBlobstoreArgs` struct) lived in `blobrepo/blob_changeset` crate, which is
obviously the most correct place for them to be located. All would be fine, had
these things been used locally only. But `RepoBlobstore` is a reasonably
widely-used type alias across our codebase and importing it from
`blob_changeset` seems weird. Let's move it into a dedicated crate.
Reviewed By: StanislavGlebik
Differential Revision: D16174126
fbshipit-source-id: b83e345adcfe567e4a67c8a1621f3a789fab63c6
Summary:
This diff does two things:
- resolves a problem with dropping censorship information when calling
`in_memory_writes_READ_DOC_COMMENT`
- prevents someone from accidentally creating a `BlobRepo` where internal blobstore's prefix is different from the `repoid`. While prefix is conceptually unrelated to a blobstore, we do care that existing blobstores continue to work, so we need this safeguard.
Reviewed By: farnz
Differential Revision: D16163225
fbshipit-source-id: fc1c9d4dc32f6958b4b0e2e61026c1f3fe5f3b17
Summary: `MemoryManifest` is not longer used anywhere
Reviewed By: farnz
Differential Revision: D16148769
fbshipit-source-id: f4e63fed3d56ade122560d271bd5701110c5dab8
Summary: This is generic implementation of tree manifest. It is going to be used to replace current hg manifest generation logic, and to implement any tree manifest bonsai derived structures like UNodes.
Reviewed By: StanislavGlebik
Differential Revision: D16048964
fbshipit-source-id: 6ffbdabca8788f22d87a61daba412cd1f9447d57
Summary: Tool that can run generation of hg manifests from bonsai, and compare them with already recorded one.
Reviewed By: StanislavGlebik
Differential Revision: D16131666
fbshipit-source-id: 8c59ec0fdf03431af85c886c22ee269cecc8e8c4
Summary:
This config option should be enabled in tests. We can use it to not run prod
services in non-test enviroment.
It will be used in the next diff to not load configerator in tests
Reviewed By: krallin
Differential Revision: D16221512
fbshipit-source-id: 7e0ba9c1d46b652a06e0e767de5df78d1671951a
Summary:
This diff sets two Rust lints to warn in fbcode:
```
[rust]
warn_lints = bare_trait_objects, ellipsis_inclusive_range_patterns
```
and fixes occurrences of those warnings within common/rust, hg, and mononoke.
Both of these lints are set to warn by default starting with rustc 1.37. Enabling them early avoids writing even more new code that needs to be fixed when we pull in 1.37 in six weeks.
Upstream tracking issue: https://github.com/rust-lang/rust/issues/54910
Reviewed By: Imxset21
Differential Revision: D16200291
fbshipit-source-id: aca11a7a944e9fa95f94e226b52f6f053b97ec74
Summary:
Move fb_status into open source fb303 and rename it to fb303_status.
Also add a getStatus() call to BaseService.
Reviewed By: wez
Differential Revision: D15800271
fbshipit-source-id: de217fb1245618d757ee853cb3ab19c7aa5dc964
Summary:
This adds support for disabling arbitrary hooks by name using a `--disable-hook` flag. This serves as a replacement for our current approach, which is to muck with configuration when we need to disable a hook on an ad-hoc basis, with mixed results ().
There are a couple use cases for which this is useful:
- Running the Hook Tailer.
- Running Mononoke Server locally.
in both cases, you probably don't want to have the verify integrity and check_unittests hooks enabled, respectively because the former might not be installed on your server (and it'll do wrong things for hook tailing anyway), and because the latter requires a token file you're not likely to have.
Reviewed By: ahornby
Differential Revision: D16201500
fbshipit-source-id: 76404eab72b9335d82235c63003c2425f0ecc480
Summary:
This updates the mononoke hook_tailer to use a MononokApp configuration. This
fixes breakage introduced in D16108456, which caused the hook_tailer to fail to
start (it was complaining that it needed configuration, but it doesn't have a
configuration).
Reviewed By: StanislavGlebik
Differential Revision: D16201501
fbshipit-source-id: 145f441c676a1ce2f0a46e1585f3715eb54db883
Summary:
Previously we used "getpack" name for both getpackv1 and getpackv2.
There were two problems with that:
1) Traffic replay doesn't support getpackv2
2) Traffic replay needs an exact name - either "getpackv1" or "getpackv2"
This diff fixes both of the problems.
Note that it'd be worth adding integration tests for traffic replay. I'll do it
in the next diffs
Reviewed By: krallin
Differential Revision: D16200293
fbshipit-source-id: 8fdb3032bcf64a10bde27bd9750e0356d889a480
Summary:
It's still clowny that we initialize CoreContext in 3 different places in
Apiserver, that's something we need to cleanup.
Reviewed By: HarveyHunt
Differential Revision: D16180999
fbshipit-source-id: 4f443e3ef25e4797500d04f5ddf6bf59e91ffa86
Summary:
This diff removes all occurrences of `extern crate` from //fbcode/common/rust// except within files included into bindgen code which are forced to use 2015 edition.
`#[macro_use]` attributes are replaced by 2018-style individual `use` of macros.
Reviewed By: jsgf
Differential Revision: D16179130
fbshipit-source-id: 25771eaec8e38043bfa1f9be267dc185758cd2d7
Summary: This change makes things same as the XDB approach.
Reviewed By: farnz
Differential Revision: D16184040
fbshipit-source-id: 0e2584943f8da98b7afbd97f2d76463d0900a8b1
Summary:
This means that we can later destructure and rebuild the `CensoredBlob` struct. If it is only
possible to construct it from a talbe name, there's no way to get this name
back from a destructured struct.
Note: I do this to eventually rebase D16163225 on top of this one, then be able implement `in_memory_writes_READ_DOC_COMMENT` in terms of destructuring of the existing `CensoredBlob`.
Reviewed By: krallin
Differential Revision: D16180745
fbshipit-source-id: e587675d444bf53f0bd25f01602503abad9d7d97
Summary:
We have problems with expensive gettreepack requests coming from quicksand.
I'll add load shedding, but before I do this I need to find what the limits for
load shedding are.
Let's use ods for that
Reviewed By: ikostia
Differential Revision: D16182208
fbshipit-source-id: 70367c876fb392f368583442ad1e45db593f9e0b
Summary:
Mine D16132403 introduced inefficiency that always fetched changesetid from
blobstore even if just id was necessary.
This diff fixes it
Reviewed By: krallin
Differential Revision: D16182140
fbshipit-source-id: 7dd4c871b8b1c19e50ba1e813d5fed42c699d064
Summary:
Report to Scuba whenever someone tries to access a blobstore which is blacklisted. Scuba reporting is done for any `get` or `put` method call.
Because of the possible overload - given the high number of requests mononoke receives and that CensoredBlobstore make the verification before we add the caching layer for blobstores - I considered reporting at most one bad request per second. If multiple requests to blacklisted blobstores are made in less than one second, only the first request should be reported. Again, this is not the best approach (to not report all of them), but performance wise is the best solution.
NOTE: I also wrote an implementation using `RwLock` (instead of the current `AtomicI64`), but atomic variables should be faster than using lockers so I gave up on that idea.
Reviewed By: ikostia, StanislavGlebik
Differential Revision: D16108456
fbshipit-source-id: 9e5338c50a1c7d15f823a2b8af177ffdb99e399f
Summary:
Seems cleaner this way. Also allows the `admin` tool to initialize
a censored blobstore.
Differential Revision: D16154919
fbshipit-source-id: f5edacc8b8332c67f1f5dfaf9bf49b4aeaecb33a
Summary:
This one is just an extraction, done to allow access to blobstore creation
without depending on the blobrepo crate (or any part thereof).
NB: the logic of censoring is still left in `blobrepo`, since that change is not a simple copy-paste.
Differential Revision: D16153024
fbshipit-source-id: b1def5f602d0c6423f28699c32f97ceae30b080c
Summary:
Sync job is used to replay pushes from Mononoke onto mercurial. It's code is in https://fburl.com/dt1hkf4g. For more about the sync job see https://fburl.com/wiki/sd2i6rch.
The goal of the sync job is to not only replicate pushes from Mononoke onto Mercurial, but also to verify that the push was correct i.e. push on Mononoke produced the same hash as on mercurial.
The problem is that we lock and unlock in two completely differen codepaths - locking is done by an external script called "failure handler" (https://fburl.com/7txm3jxf), while unlocking is a part of the sync job.
Ideally we'd like to lock and unlock in the sync job itself and remove "failure handler" entirely.
Reviewed By: StanislavGlebik
Differential Revision: D16107759
fbshipit-source-id: a418f8d0f48fa6db82476be72a91adbc03b66168
Summary:
Because of the recursion this function can cause stackoverflow if there are a
lot of commits that don't have hg changeset generated.
Let's rewrite it using loop_fn.
Note that there's "panic!()" in fetch_hg_changeset_from_mapping. While having
panics in the code is not great, there doesn't seem to be a way around it. When
generating new hg changesets it expects that parents' changesets are already in
the repository. If they are not generated then it's an internal logic error,
and in that case it's probably better to panic.
Reviewed By: krallin
Differential Revision: D16132403
fbshipit-source-id: 17e48ecce653f72e3f576f3698330d90a4dd7902
Summary: With these fixes running the healer for `repo_id` 502 seems to work.
Reviewed By: aslpavel
Differential Revision: D16140872
fbshipit-source-id: bf2caf7428a74cf14cd61ea8b52b7e5822432646
Summary: It's a footgun, as its memory use is bounded by the size of the repo, so it's almost never what you really want, but instead a band-aid over a deeper problem.
Reviewed By: StanislavGlebik
Differential Revision: D16132084
fbshipit-source-id: 12683801652509ebb09881595768d640df0b7490
Summary:
Previously:
```
error[E0599]: no method named `boxify` found for type `futures::future::result_::FutureResult<_, _>` in the current scope
--> experimental/dtolnay/thttp/main.rs:23:21
|
23 | let transport = try_boxfuture!(HttpClient::new(ENDPOINT));
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= help: items from traits can only be used if the trait is in scope
= note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)
help: the following trait is implemented but not in scope, perhaps add a `use` for it:
|
1 | use futures_ext::FutureExt;
|
```
Reviewed By: Imxset21
Differential Revision: D16134586
fbshipit-source-id: 324d2c7223c5e71ffc1f4390cd8fc0b488243c00
Summary: Gettreepack requests can be slow if there's no base manifest. Instead of trying to track every manifest we've passed down to the client (which is CPU and memory expensive, as we end up tracking every `(path, manifesthash)` pair in the output), simply use the first manifest we were asked for as the base manifest in this case
Reviewed By: StanislavGlebik
Differential Revision: D16120603
fbshipit-source-id: 19119e481dfad37c2539f6f02f6ffccb13bc9895
Summary: Log when a `censored file` is accessed via calling `get` or `put` method.
Reviewed By: StanislavGlebik
Differential Revision: D16089138
fbshipit-source-id: 0d2f0e21e7afcad8783be7587e6b676af20ba029
Summary:
When the underlying stream in a `BytesStream` hits EOF, and the `BytesStream` is asked for more data, it'll busy loop until more data becomes available.
This happens because when the `BytesStream` hits EOF on the underlying stream, it'll record that the stream is done, and won't ever poll the underlying stream again in `poll_buffer`. However, in `poll_buffer_until`, we essentially busy loop through calls to `poll_buffer`
When that happens, the process goes into an infinite loop of:
- Checking that it doesn't have enough bytes.
- Observing that the stream is done.
- Calling `poll_buffer` and not making any progress.
Instead, the right behavior would be to return a successful read of length zero in `read()` to indicate EOF to the caller. This is what this patch does.
It's worth noting that even if the underlying stream returned more data after it reported it was exhausted (callers should avoid polling them again ... or use `fuse()`), the `BytesStream` won't ever poll for that data, since it skips the poll because `stream_done` is set.
Reviewed By: farnz
Differential Revision: D16130672
fbshipit-source-id: a61c39feb1aa1ac74bbef909e47d698051477533
Summary:
1. Let's import `PartId` rather than redefine it
2. Let's extract chainig boilerplate into an extra fn
Reviewed By: StanislavGlebik
Differential Revision: D16122281
fbshipit-source-id: eb1f8b9f3c3440acb9106377d978090389d6c66c
Summary:
```
pub fn top_level_launch<F>(future: F) -> Result<F::Item, F::Error>
where
F: Future + Send + 'static,
F::Item: Send,
F::Error: Send;
```
Starts the Tokio runtime using the supplied future to bootstrap the execution.
### Similar APIs
This function is equivalent to [`tokio::run`](https://docs.rs/tokio/0.1.22/tokio/fn.run.html) except that it also returns the future's resolved value as a Result. Thus it requires `F::Item: Send` and `F::Error: Send`.
This function has the same signature as [`Future::wait`](https://docs.rs/futures/0.1.28/futures/future/trait.Future.html#method.wait) which also goes from `F: Future` -> `Result<F::Item, F::Error>`, but `wait` requires an ambient futures runtime to already exist.
### Details
This function does the following:
- Start the Tokio runtime using a default configuration.
- Spawn the given future onto the thread pool.
- Block the current thread until the runtime shuts down.
- Send ownership of the future's resolved value back to the caller's thread.
Note that the function will not return immediately once the original future has completed. Instead it waits for the entire runtime to become idle.
### Panics
This function panics if called from the context of an executor.
Reviewed By: bolinfest
Differential Revision: D16116198
fbshipit-source-id: 52009e17c1ccc566e0c156a86f8fd5844e0e2491
Summary:
`local_instances` option was used to create fileblobstore or sqlite blobstore.
Now we use mononoke config for this purpose. Since this option is no longer
useful let's delete it
Reviewed By: krallin
Differential Revision: D16120065
fbshipit-source-id: 375a168b27e7f2cf1a6a77f487c5e013f9004546
