Summary:
Edenapi lookup (for file content, filenodes and trees): check all the multiplexed blobstores when we check is_present.
This will help us to avoid undesired behaviour for commit cloud blobs that haven't been replicated to all blobstores. Healer currently doesn't check commit cloud blobs.
Reviewed By: StanislavGlebik
Differential Revision: D30839608
fbshipit-source-id: d13cd4500f7b14731d8b75c763c14a698399ba02
Summary:
Some time ago (see D25910464 (fca761e153)) we've started using Background session class
while deriving data. This was done to avoid overloading blobstore sync queue - if Background
session class is set then multiplex blobstore waits for all blobstores to
finish instead of writing to the blobstore sync queue right away. However if any of the
blobstores fails then we start writing to the blobstore sync queue. In theory it should have avoided the problem of overloading blobstore sync queue while having the same multiplex reliability (i.e. if only a single blobstore fails the whole multiplex put doesn't fail)
Unfortunately there was a flaw - if blobstore put of a single blobstore wasn't
failing but was just too slow, then the whole multiplexed put operation becomes
too slow. This diff fixes this flaw by adding a timeout - if multiplexed put is
taking too long then we fallback to writing entries to the blobstore sync
queue.
Note that I added a new session class - BackgroundUnlessTooSlow -
because I figured that in some cases we are ok with waiting a long time but not
writing to the sync queue. Skiplist builder might be a good example of that -
since it's doing overwrites then we don't want to write to the blobstore sync
queue at all, because healer doesn't process overwrites correctly.
Reviewed By: farnz
Differential Revision: D30892377
fbshipit-source-id: 69ac1795002b124e11daac13d8bfe59895191168
Summary:
Add impls for Layer for Box/Arc<L: Layer> and <dyn Layer>. Also a pile of other
updates in git which haven't been published to crates.io yet, including proper
level filtering of trace events being fed into log.
Reviewed By: dtolnay
Differential Revision: D30829927
fbshipit-source-id: c01c9369222df2af663e8f8bf59ea78ee12f7866
Summary:
Like it says in the title, this updates us to use Daemonize 0.5, though from
Github and not Crates.io, because it hasn't been released to the latter yet.
The main motivation here is to pull in
https://github.com/knsd/daemonize/pull/39 to avoid leaking PID files to
children of the daemon.
This required some changes in `hphp/hack/src/facebook/hh_decl` and `xplat/rust/mobium` since the way to
run code after daemonization has changed (and became more flexible).
Reviewed By: ndmitchell
Differential Revision: D30694946
fbshipit-source-id: d99768febe449d7a079feec78ab8826d0e29f1ef
Summary:
Manual component version update
Bump Schedule: https://www.internalfb.com/intern/msdk/bump/?schedule_fbid=342556550408072
Package: https://www.internalfb.com/intern/msdk/package/181247287328949/
Oncall Team: rust_foundation
NOTE: This build is expected to expire at 2022/09/01 09:14AM PDT
---------
New project source changes since last bump based on D30663071 (08e362a355e0a64a503f5073f57f927394696b8c at 2021/08/31 03:47AM -05):
| 2021/08/31 04:41AM -05 | generatedunixname89002005294178 | D30665384 | [MSDK] Update autocargo component on FBS:master |
| 2021/08/31 07:14PM PDT | kavoor | D30681642 | [autocargo] Make cxx-build match version of cxx |
| 2021/09/01 04:05PM BST | krallin | D30698095 | autocargo: include generated comment in OSS manifests |
---------
build-break (bot commits are not reviewed by a human)
Reviewed By: farnz
Differential Revision: D30717040
fbshipit-source-id: 2c1d09f0d51b6ff2e2636496cf22bcf781f22889
Summary:
The mockall crate's `automock` attribute previously created nondeterministic output, which leads to frequent random "Found possibly newer version of crate" failures in Buck builds that involve cache.
The affected trait in Conveyor is:
https://www.internalfb.com/code/fbsource/[4753807291f7275a061d67cead04ea12e7b38ae2]/fbcode/conveyor/common/just_knobs/src/lib.rs?lines=13-23
which has a method with two lifetime parameters. Mockall's generated code shuffled them in random order due to emitting the lifetimes in HashSet order. The generated code would randomly contain one of these two types:
`Box<dyn for<'b, 'a> FnMut(&str, Option<&'a str>, Option<&'b str>) -> Result<bool> + Send>`
`Box<dyn for<'a, 'b> FnMut(&str, Option<&'a str>, Option<&'b str>) -> Result<bool> + Send>`
Reviewed By: jsgf
Differential Revision: D30656936
fbshipit-source-id: c1a251774333d7a4001a7492c1995efd84ff22e5
Summary:
This config option was used to slowly roll out LFS for a repo.
However, it is no longer used and can therefore be removed.
Reviewed By: StanislavGlebik
Differential Revision: D30511880
fbshipit-source-id: 59fe5925cc203aa609488fdf8ea29e9ff65ee862
Summary:
If hg sync job is pushing a commit, then we have no choice but to accept it,
even if it's too big. So let's not fail if too many commits are pushed.
Reviewed By: ahornby
Differential Revision: D30535025
fbshipit-source-id: eb607a8fbd691d6591ad990e0920411b1ad2f09c
Summary:
The diff is giant, but it's just a one-line change to add the
nested-values feature to slog, we just have a whole bunch of projects dependent
on slog.
Reviewed By: dtolnay
Differential Revision: D30351289
fbshipit-source-id: b6c1c896b06cbdf23b1f92c0aac9a97aa116085d
Summary:
Autocargo only allows 1 rust-library per Cargo.toml, but right now we have 3
per Thrift library so that doesn't work:
https://www.internalfb.com/intern/sandcastle/log/?instance_id=27021598231105145&step_id=27021602582167211&step_index=13&name=Run%20config
There's little benefit in Autocargo-ifying those rules anyway since they're of
use to Thrift servers and this doesn't work at all in our OSS builds, so let's
just see if we can just noop them. That'll make the crate not exist at all as a
dep, but even considering that it exists only to link to a C++ library that
Autocargo doesn'tk now how to build anyway, that seems OK?
drop-conflicts
Reviewed By: markbt
Differential Revision: D30304720
fbshipit-source-id: 047524985b2dadab8610267c05e3a1b3770e84e6
Summary: Previous diffs switched all our usage from chashmap to dashmap as dashmap upstream is more responsive. Now remove chashmap from the cargo vendoring.
Reviewed By: dtolnay
Differential Revision: D30046522
fbshipit-source-id: 111ef9375bd8095f8b7c95752ecbc1988fb0438d
Summary:
The patches to these crates have been upstreamed.
allow-large-files
Reviewed By: jsgf
Differential Revision: D29891894
fbshipit-source-id: a9f2ee0744752b689992b770fc66b6e66b3eda2b
Summary:
I noticed that reading one of the mononoke configs was failing with
```
invalid type: string \"YnrbN4fJXYGlR1EzoxLRvVbibyUiRM/HZThRJnKBThA\", expected
a sequence at line 2587 column 61)\x18ninvalid type: string
\"YnrbN4fJXYGlR1EzoxLRvVbibyUiRM/HZThRJnKBThA\", expected a sequence at line
2587 column 61
```
The problem is coming from the fact that configerator configs use thrift simple
json encoding, which is different from normal json encoding. At the very least
the difference is in how binary fields are encoded - thrift simple json
encoding uses base64 to encode them. [1]
Because of this encoding difference reading the configs with binary fields in
them fails.
This diff fixes it by using simple_json deserialization for
get_config_handle()... but the existing callers used the old broken
`get_config_handle()` which is
incompatible with the new one. Old `get_config_handle()` relied on the fact
that serde::Deserializer can be used to deserialize the config, while thrift
simple json doesn't implement serde::Deserializer.
As a first step I migrated existing callers to use old deprecated method, and
we can migrate them to the new one as needed.
[1] It was a bit hard to figure out for sure what kind of encoding is used, but
discussion in
https://fb.workplace.com/groups/configerator.users/posts/3062233117342191
suggests that it's thrift simple json encoding after all
Reviewed By: farnz
Differential Revision: D29815932
fbshipit-source-id: 6a823d0e01abe641e0e924a1b2a4dc174687c0b4
Summary: Currently we only print what's in the `error` annotation.
Reviewed By: krallin
Differential Revision: D29794843
fbshipit-source-id: a2c411208d7be8fd856dd9b3f82fd96a4ed37aee
Summary:
Now that Mononoke uses the `rate_limiting` library we can shed load if
a server is overloaded. Add load shedding checks to the entry points for
wireproto and EdenAPI HTTP traffic.
At the time of writing, there aren't any load shedding limits configure so this
change won't have any effect.
Differential Revision: D29396504
fbshipit-source-id: c90cc40fc2609bdae1a267be3a1aecfe7fd33b7b
Summary:
Update Mononoke server to use the new `rate_limiting` crate. This diff
also removes the old rate limiting library.
Differential Revision: D29396507
fbshipit-source-id: 05adb9322705b771a739c8bcaf2816c95218a42d
Summary:
This reads the config added on D29305462. It populates it into `CommonConfig` struct, and also adds it to `RepoFactory`, but doesn't yet use it anywhere, this will be done on the next diff.
There is a single behaviour change in this diff, which I believe should be harmless but is noted in the comments in case it isn't.
Reviewed By: markbt
Differential Revision: D29272581
fbshipit-source-id: 62cd7dc78478c1d8cb212eafdd789527ead50ef6
Summary:
This is just updating the os_info crate to my fork with a fix for Centos
Stream: https://github.com/stanislav-tkach/os_info/pull/267
Reviewed By: quark-zju
Differential Revision: D29410043
fbshipit-source-id: 3642e704f5a056e75fee4421dc59020fde13ed5e
Summary: There is a regression in 1.7.0 (which we're on at the moment) so we might as well update.
Reviewed By: zertosh, farnz
Differential Revision: D29358047
fbshipit-source-id: 226393d79c165455d27f7a09b14b40c6a30d96d3
Summary:
Pull in a patch which fixes writing out an incorrect entsize for the
`SHT_GNU_versym` section:
ddbae72082
Reviewed By: igorsugak
Differential Revision: D29248208
fbshipit-source-id: 90bbaa179df79e817e3eaa846ecfef5c1236073a
Summary: Update versions for several of the crates we depend on.
Reviewed By: danobi
Differential Revision: D29165283
fbshipit-source-id: baaa9fa106b7dad000f93d2eefa95867ac46e5a1
Summary: revert the zstd crates back to previous version
Reviewed By: johansglock
Differential Revision: D29038514
fbshipit-source-id: 3cbc31203052034bca428441d5514557311b86ae
Summary: Update to latest version. This includes a patch to async-compression crate from [my PR updating it](https://github.com/Nemo157/async-compression/pull/125), I will remove once the crate is released.
Reviewed By: mitrandir77
Differential Revision: D28897019
fbshipit-source-id: 07c72f2880e7f8b85097837d084178c6625e77be
Summary:
Time 0.2 is current, and 0.1 is long obsolete. Unfortunately there's a
large 0.1 -> 0.2 API change, so I preserved 0.1 and updated the targets of its
users. Also unfortunate that `chrono` has `oldtime` as a default feature, which
makes it use `time-0.1`'s `Duration` type. Excluding it from the features
doesn't help because every other user is specifying it by default.
Reviewed By: dtolnay
Differential Revision: D28854148
fbshipit-source-id: 0c41ac6b998dfbdcddc85a22178aadb05e2b2f2b
Summary:
In practice, if the client has disconnected it's unlikely that the thing that
will fail is a flush (instead if will probably be the poll that checks if we
can write before). So, let's track that separately.
This isn't super important since we can already infer the timing from when the
error was logged but it'll make the logs less ambiguous.
Reviewed By: johansglock
Differential Revision: D28637258
fbshipit-source-id: 3bc9c9aaa9fc8cf7a2d2514fb520cb1729f4c560
Summary:
Add streaming compression support to Mononoke. We do the compression inside our
own framing, to make sure the compression doesn't interfere with any buffering,
flushing and keepalives.
Reviewed By: krallin
Differential Revision: D28323591
fbshipit-source-id: 7192bedcf12a2d0ec025deed8b4fa3857eefd508
Summary:
When you spawn a task, Tokio puts it on a "LIFO slot" associated with the
current thread. While the task is in the LIFO slot, it is not eligible to be
run by other threads.
If the thread that just spawned `fwd` above goes and does some expensive
synchronous work in `request_handler` (we'd like to avoid that but sometimes
that happens), then that will delay `fwd`.
This means that notably keepalives will not be sent (you can repro by putting a
`std:🧵:sleep` right after we spawn `fwd`). To mitigate this, we spawn
another dummy taks here. This task will take `fwd`'s place in the LIFO slot,
thus pushing `fwd` onto a task queue where other runtime threads can claim it.
This way, even if this thread goes do some expensive CPU-bound work, we won't
delay keepalives.
This doesn't solve the fact that we probably shouldn't be doing such CPU-bound
work to begin with, but at least it might mitigate the issue such that we don't
have I/O delayed even though the runtime isn't visibly overloaded.
Reviewed By: johansglock
Differential Revision: D28412114
fbshipit-source-id: b56c0156ac6cf991cc899a82e3d2a96c63216fda
Summary:
We have clients mysteriously timing out despite the fact that we should
normally be sending keepalives. To understand this, add some logging so that if
a client disconnects, we log a lot more detailed information to Scuba.
Reviewed By: mitrandir77
Differential Revision: D28383122
fbshipit-source-id: 0e49262bcc08c75f8e06eae17c882742b58ea51d
Summary:
I'd like to have some socket level logging, and it's good to capture the
metadata there so that we have client identities, session ids, etc.
But I can't do this if I don't have the metadata! This updates our code so that the
metadata is available where this logging will go.
This means we can't rely on loggers in `try_convert_preamble_to_metadata`, but
given this is now only used for hgcli, I don't think that matters, so I just removed it.
Reviewed By: mitrandir77
Differential Revision: D28383121
fbshipit-source-id: 27532f021a9082e74403bba73cad4fc6d0a6d97c
Summary:
I'd like to have some socket level logging, and it's good to capture the
metadata there so that we have client identities, session ids, etc.
To do so I need to not move it into the session. This does that.
Reviewed By: mitrandir77
Differential Revision: D28383123
fbshipit-source-id: 3fd10c3720465824dbcb2528227cbb3521d3068a
Summary: Log the blobstore id as part of sampled pack info. This is allows running the walker pack info logging directly agains a multiplex rather than invoke it for one component at a time.
Reviewed By: farnz
Differential Revision: D28264093
fbshipit-source-id: 0502175200190527b7cc1cf3c48b8154c8b27c90
Summary:
This is going to enable the background update in SegmentedChangelog to log
entries to Scuba.
The scuba sample builder is not fundamentally different than other elements of
the environment. It is used slightly differently to, for example, Logger,
because it has to cloned in all places that want to log rows but otherwise it
has the same characteristics.
Reviewed By: krallin
Differential Revision: D28210008
fbshipit-source-id: 68468868d13f29dddf21095bd7526cb4ff690786
Summary:
Rename `new_with_containers` to `new`. `new` the standard name for
what the fuction does.
Reviewed By: krallin
Differential Revision: D28210005
fbshipit-source-id: 9ad0dad5e8afd89d7538fd32d93de10837e26f9d
Summary:
Refactoring the construction logic for CoreContext.
`test_mock_class` is used in only one place. That usage can be replaced with
`test_mock_session`.
Reviewed By: krallin
Differential Revision: D28210007
fbshipit-source-id: c81ca31cb7255ef640fe641e9b38f239cfb69267
Summary:
Refactoring CoreContext construction to express purpose. We will have
a constructor for request processing, for background processing.
Bulk processing is another category that has it's own constructor
already. Renaming it to make it more prominent.
Reviewed By: krallin
Differential Revision: D28210006
fbshipit-source-id: 2bb74d97e2f3588aa539e58c3d6dd6842f898121
Summary: Upstream crate has landed my PR for zstd 1.4.9 support and made a release, so can remove this patch now.
Reviewed By: ikostia
Differential Revision: D28221163
fbshipit-source-id: b95a6bee4f0c8d11f495dc17b2737c9ac9142b36
Summary:
Quicksand is about 60% of our logging, but we don't really need to log
everything it does because the workload is pretty much the same across all
hosts.
Reviewed By: HarveyHunt
Differential Revision: D28225416
fbshipit-source-id: 68c0f68c4da73e0f7543754c917d7efa4bbf374e
