Summary:
Cargo builds run multiple tests in same process, which was causing a failure since MononokeMatches started doing the one time log::set_boxed_logger() call.
I made MononokeMatches:new() error rather than panic in this case as it was already returning a Result anyway.
The failing test was only testing a very small section of code in block_execute, so this change removes it (its covered by integration test). It also gave some MononokeMatches coverage as a side effect (also covered by integration test).
Reviewed By: krallin
Differential Revision: D27891187
fbshipit-source-id: 9787029b610040cbf0125ab79748d6a3e540d2ae
Summary: Remove gotham and hyper patches as referenced PRs have been released
Reviewed By: krallin
Differential Revision: D27905248
fbshipit-source-id: a2b25562614b71b25536b29bb1657a3f3a5de83c
Summary:
We want to distinguish user vs system errors in `configo_client` and its users
(`mononoke_config` for instance). The reason is to allow `scs_server`
distinguish the two types of errors. Normally user errors would only ever be
instantiated fairly "shallowly" in the `scs_server` itself, but `configo_client`
is a transactional client (by this I mean that it calls user-provided transformation functions on fetched data), so we need to allow for a user error to originate from these updater functions.
Reviewed By: StanislavGlebik
Differential Revision: D27880928
fbshipit-source-id: e00a5580a339fdabc4b858235da9cd7e9fc7a376
Summary:
hg.py has a hard coded list of what configs it will copy from the source repo to the remote repo object (hg.py remoteui())
we use one of lfs and one of infinitepush option in edenscm/hgext/clienttelemetry.py where the remote repo object is used
Reviewed By: ahornby
Differential Revision: D27906275
fbshipit-source-id: d551934437126fdd0b920354bf4c51a6e09bafb2
Summary:
Integration test showing that MismatchedHeads errors are sent over the wire to
the client.
Reviewed By: quark-zju
Differential Revision: D27798555
fbshipit-source-id: b14a213e9055486bf07ecbb4b5453385df701b48
Summary:
This error occurs when the client sends us a set of heads that we cannot match.
For example when the client forces a commit in the master group; this was
possible with revlogs but should be a bug with Segmented Changelog. This can
also happen when master moves backwards, clients and server have multiple heads
then the server reseeds.
Clients that get this error should reclone.
Reviewed By: quark-zju
Differential Revision: D27786602
fbshipit-source-id: 9854ccee929ae0a845236ebd83ed68158c93fc7a
Summary:
We want to distinguish between no location and failure to compute location.
It is useful to know on the client side if we failed to compute the location of
some hash or we are not sending it because we don't have a location for it.
We have different options for hash-to-location in particular but the problem is
a general one. Does it make sense for us to skip sending error at the EdenApi
handler layer? I don't think so, I think that's an old hask. We should be
evolving the EdenApi handler layer to handle application errors.
This is still at the exploration phase. I haven't fixed the client side to
handle the errors.
Reviewed By: quark-zju, kulshrax
Differential Revision: D27549926
fbshipit-source-id: 7094160fe997af0e69c8d006b9731ea7dfb3e3f8
Summary:
The goal here is to add an easy way of propagating application errors from
server to the client.
The most convenient form for an error message is a message string so we just
start with that. I think the most important thing to add is some way of
communicating whether the error is retryable or not.
With conversions for anyhow::Error to WireError and from WireError to
ServerError it should be trivial for application code to pass application
errors down to the client.
The format here is driven by the fact that we have streams in the response
object. A batch oriented format for responses has more options. For example
with batches it is common to have a response object that has 3 categories:
1. found: HashMap, 2. not_found: Vec, 3. errors: Vec/HashMap
Reviewed By: kulshrax
Differential Revision: D27549923
fbshipit-source-id: 33b790253adc4761ea9ac7caced4148f4026e620
Summary:
On Mac this introduces 150ms delays in every indexedlog flush. During
an amend of a stack with 32 commits, this fsync accounted for 2/3rds of the time
spent.
Since commitcloud is pretty reliable these days, I think this is no longer
necessary.
Reviewed By: andll
Differential Revision: D27896589
fbshipit-source-id: a13a494c54ffea5a670ed942b366620619af2bd0
Summary: This is needed for rendering multiple progress bars in `cmd.exe`.
Reviewed By: andll
Differential Revision: D27887601
fbshipit-source-id: 9aa987cef327de91408f2e38b4d2e551fe10e39b
Summary:
`distutils_rust` tries to use `mt` to merge the "long path aware" Windows
manifest. It tries to deal with 2 cases:
- The exe already has an embedded manifest. Use `-inputresource` to merge them.
- The exe does not have an embedded manifest. Do not use `-inputresource` since
it will cause an error.
It does so by first passing `-inputresource`, then drop it if the error message
contains "The specified resource type cannot be found in the image file".
However, the error message is translated. Depending on system locale setting,
`mt` might output:
mt : general error c101008c: Failed to read the manifest from the resource of file "hg.exe". The specified resource type cannot be found in the image file.
mt : general error c101008c: Failed to read the manifest from the resource of file "hg.exe". 指定的映像文件不包含资源区域。
So let's check the error code `c101008c` instead.
Reviewed By: andll
Differential Revision: D27885096
fbshipit-source-id: 3d42a3b88ae593434e7104c7899bf6b4f3366180
Summary: `CaseSensitivity::Sensitive` is better than a mere `true` out of nowhere.
Reviewed By: kmancini
Differential Revision: D27867180
fbshipit-source-id: 39d21d3cc3b70c78c6984c3ddbd65b53520770be
Summary:
This is a baseline packer, to let us experiment with packing.
It chooses the dictionary that gives us the smallest size, but does nothing intelligent around key handling.
Note that at this point, packing is not always a win - e.g. the alias blobs are 432 bytes in total, but the resulting pack is 1528 bytes.
Reviewed By: ahornby
Differential Revision: D27795486
fbshipit-source-id: 05620aadbf7b680cbfcb0932778f5849eaab8a48
Summary:
Let's make sure filenodes are warmed at the same time as hg changesets.
We do it implicitly by deriving them in getbundle, however it results in more
attempts of deriving filenodes, because we'd try to derive on every pull.
This is not a huge problem, but that would mean that we could send more
requests to the root filenode to check whether filenodes for a commit were derived or not.
And I remember looking at the logs a few days ago and seeing a few filenodes errors in our
scuba table, which I suspected might be related to the fact that filenodes are
not in derived data cache.
There's a high chance i'm wrong about the root cause of these problems this change should be
reasonable to do anyway.
Reviewed By: krallin
Differential Revision: D27822519
fbshipit-source-id: c878206a09bd19e4a68327dddf27aae10490740f
Summary:
Instead of using get_public method which queries bookmarks let's call
get_public_raw instead, which just does a phases fetch from a local db.
See previous diff for more motivation
Reviewed By: krallin
Differential Revision: D27821547
fbshipit-source-id: a71c8c9ad283259e9be98e63c9c72428e35c6142
Summary: `%s` with a rev number is not accepted in the current codebase. Use `%d` instead.
Reviewed By: ikostia
Differential Revision: D27874102
fbshipit-source-id: b83c40b7182da8639e82bea7bd00a036be4120c4
Summary: `%s` with a rev number is not accepted in the current codebase. Use `%d` instead.
Reviewed By: ikostia
Differential Revision: D27873899
fbshipit-source-id: b34eb0b80f0789c9e06af366bfdaa884c5c69357
Summary: `%s` with `revid` is not accepted in the current codebase.
Reviewed By: ikostia
Differential Revision: D27873898
fbshipit-source-id: e3790855892d3b07e1e5ea6bd92a14738bf6c100
Summary:
We didn't log it to perf counters log, and that makes it hard to aggregate,
show distributions etc
Let's start doing that
Reviewed By: krallin
Differential Revision: D27856968
fbshipit-source-id: 82fbba70154ee011073f3122256bd296bbb938ae
Summary: Its more efficient to bulk load the mappings for a chunk than do the queries one by one
Differential Revision: D27801830
fbshipit-source-id: 9c38ddfb1c1d827fc3028cd09f9ad51e3cbee5dc
Summary: Add an accessor so that we keep a reverse mapping of the WalkState::bcs_to_hg member as a cache of bonsai to hg mappings and also populate it on derivations.
Differential Revision: D27800533
fbshipit-source-id: f9b1c279a78ce3791013c3c83a32251fdc3ad77f
Summary: Add an accessor so that we can use the WalkState::hg_to_bcs member as a cache of hg to bonsai mappings
Reviewed By: farnz
Differential Revision: D27797638
fbshipit-source-id: 44322e93849ea78b255b2e3cb05feb8db6b4c7a7
Summary: This diff makes treeoverlay the default overlay type for Windows users.
Reviewed By: kmancini
Differential Revision: D27247658
fbshipit-source-id: 866eafc794eff1c262eab3061f14eb597bea0a66
Summary: This diff allows EdenFS to create tree overlay based on checkout configuration.
Reviewed By: kmancini
Differential Revision: D27242580
fbshipit-source-id: d0ebe33017c16517c117c1886f62bc9c6fe9f466
Summary:
`debugresetheads` is expected to remove all non-essential heads. That
includes bookmarks.
Reviewed By: kulshrax
Differential Revision: D27861548
fbshipit-source-id: 045976a5a9e27e7eee7ee48448c44552da439983
Summary:
Now that lastCheckoutTime is a single uint64_t, we no longer need a lock to
protect it, simple atomics are sufficient. Since reading an atomic usually
doesn't require any atomic operation, this will save a handful of atomics when
loading an inode where the last checkout time is read.
Reviewed By: chadaustin
Differential Revision: D27860653
fbshipit-source-id: 464e950c949ca243664d213da99d96ff5d0fcbb8
Summary:
The lastCheckoutTime is mostly used to initialize the timestamps of newly
loaded inodes and since these store an EdenTimestamp, we incur a conversion for
every inode loads. Instead of doing this conversion, we can simply make it an
EdenTimestamp directly.
Similarly, the getNow function was always converted to an EdenTimestamp
(sometimes more than once), we can also make it return the EdenTimestamp
directly.
Reviewed By: chadaustin
Differential Revision: D27860652
fbshipit-source-id: 9ea8fe9a312e6c3d8667b93130bb32a46ab92961
Summary:
Some test runner don't properly redirect stdout/stderr of nested processes, or
even direct writes to filedescriptors. On these, debugging a test failure is
almost impossible for EdenFS as we rely on the test output to be interleaved
with the EdenFS logs to understand what the daemon is doing.
To solve this, we can simply create a thread that redirects the output of
EdenFS to sys.std{out,err}.
Reviewed By: kmancini
Differential Revision: D27570966
fbshipit-source-id: 6a8d5229d8d5d141e6ab423f7658744b42af46e3
Summary: The Python `[auth]` matching code does not take cert validity into account when performing certificate matching, whereas the Rust version of the code does. In practice, the existing call sites for the Rust code disable match-time validation, and instead validate the certificate at time-of-use. This diff makes the Rust code's behavior match Python so we can remove the latter entirely.
Reviewed By: DurhamG
Differential Revision: D27837343
fbshipit-source-id: 0bfb5ebc3a36c8fa748cb289e2d8d1495ba8b296
Summary:
The svfs might have a different permission setup (ex. g+s, on ext4) that cannot
be applied to other vfs (ex. on edenfs). Do not inherit it. Instead, calculate
proper mode from the vfs root (ex. `.hg`).
Practically, the `createemode` is `None` in most of our repos. However,
`debugsegmentclone` might create svfs with `g+s` mode due to indexedlog's
`mkdir -p` recursively chmods created parents.
The original logic was added in 6590bef21 (FBS), 80e51429cb9a (HG) in 2008 with
little review comments: https://www.mercurial-scm.org/pipermail/mercurial-devel/2008-July/007269.html
Reviewed By: DurhamG
Differential Revision: D27860581
fbshipit-source-id: 43f93080621aaef168d2cecae46fd6dfb879fa1c
Summary: Enables the `Serialize` and `Deserialize` impls on the `Uuid` type.
Reviewed By: dtolnay
Differential Revision: D27799952
fbshipit-source-id: 4b0e2f8ab4ede20a2113fc1dda42c2ba8b3d0b35
Summary:
Previously we were always caching bookmarks, but D27323369 (f902acfcd1) accidentally remove
that. Let's add it back
Reviewed By: krallin
Differential Revision: D27859523
fbshipit-source-id: 8137c838fc56ecbbc64ba139d4a590dccd011bbc
Summary:
I am debugging why some people get vim to pop up during a merge conflict and some do not.
also fixes a few lint issues
Reviewed By: DurhamG
Differential Revision: D27684419
fbshipit-source-id: f636d71c18353a3816d1e442c05790cf4fd7b90b
Summary: I am removing this change because we've decided to store prepushrebase changeset id server-side.
Reviewed By: ikostia
Differential Revision: D27853518
fbshipit-source-id: 888897bc48c67477309b09af5f8c1825ce20cbca
Summary:
To prevent bonsai changeset divergence between prod and backup repo by copying
bonsais from prod repo directly during hg sync job push.
See more details about motivation in D27824210
Reviewed By: ikostia
Differential Revision: D27852341
fbshipit-source-id: 93e0b1891008858eb99d5e692e4dd60c2e23f446
Summary:
In the next diff it's going to be used to copy bonsais from the prod repo
during hg sync job, and in this diff I move this code to the common place so
that we can use it in the next diff
Differential Revision: D27852340
fbshipit-source-id: 9744571430e15a9d7f1e569d9b6690bc45787bd2
Summary:
This is not used on its on, but in subsequent diffs I will add a use-case, by
the megarepo configs crate.
When built in non-fbcode mode, this crate does not export anything. I chose this approach as opposed to the approach of exporting no-op stubs to force the clients to pay attention and implement gating their side too. This seems reasonable for a rather generic configo client.
Reviewed By: StanislavGlebik
Differential Revision: D27790753
fbshipit-source-id: d6dcec884ed7aa88abe5796ef0e58be8525893e2
Summary:
This diff does the following:
1. makes `megarepo_add_sync_target` into an async call
2. wraps all the async call responses into an additional struct to convey the
idea of pending requests
3. fixes code which imports/implements these interfaces
1 is needed because adding a new target will need to create an initial state of
the megarepo (so not just write some configs), and that is an expensive
operation.
2 is needed because we don't want to express the idea of "this request is not
yet processed" through a thrift exception. Instead, let make `_poll` calls
return a struct with a single optional field. When present, that field will
contain the payload of response to the underlying request. When absent, it will
indicate the fact that the request is still pending.
Of course, this is a compatibility-breaking change, that's why I want to get it in as early as possible (while there are no real clients calling changed methods).
Reviewed By: StanislavGlebik
Differential Revision: D27823377
fbshipit-source-id: dc2a5ed327b38d1cacd575af9d7edf5768f9c377
Summary: Now we're on rustc 1.51 the fork is no longer needed.
Reviewed By: dtolnay
Differential Revision: D27827632
fbshipit-source-id: 131841590d3987d53f5f8afb5ebc205cd36937fb
Summary:
If you ask for a range starting at byte 3, and the chunks are of size 3, then
you don't actually need need the first chunk, but right now we'll fetch it
then extract zero bytes from it.
This is quite wasteful, and for LFS range fetches will be problematic since it
basically doubles the volume of stuff we need to keep in cache (we need both
chunks).
Reviewed By: farnz
Differential Revision: D27824411
fbshipit-source-id: 7103f5b4d5bb78f023245f3e8a1bcb0c2f28faab
Summary:
Like it says in the title. We'd like to ask for the exact size that was
configured, because this way we can set the chunk size to the LFS threshold and
it avoids overlapping any file chunks server side.
Reviewed By: DurhamG
Differential Revision: D27824418
fbshipit-source-id: 43f40eb87080ec58e813ba1f1dda5b6a5e9f98ee
Summary:
While soft mount are nice as they allow the server (edenfs) to die and the
client applications to not end up in D state, this also force a maximum
(non-configuerable) 60s timeout for all IOs, after which application receive a
ETIMEDOUT. Thus, we need to not make the mount hard, thankfully, since the
mount is INTR, applications should not stay in D state if EdenFS dies.
Reviewed By: genevievehelsel
Differential Revision: D27808311
fbshipit-source-id: 17c30e88e5b236418064d8c309d85fdc6f1ca3e9
Summary:
Like it says in the title, this includes rendezvous into changesets. This is
our busiest connection by far, and it is often hitting the limits of our
connection pool: https://fburl.com/ods/kuq5x1vw
Reviewed By: markbt
Differential Revision: D27794574
fbshipit-source-id: e2574ce003f12f6c9ecafd0079fe5194cc63c24b
Summary:
I'd like to add RendezVous here (because this is our busiest connection:
https://fburl.com/ods/6d4a9qb5), and it'll be easier to do so if I just have
one code path to change instead of two.
Reviewed By: farnz
Differential Revision: D27794575
fbshipit-source-id: 350e3f8e3f3a74cb7c675cef1264c8083c516480
Summary:
After doing some local benchmarking (using MononokeApi instantiation as the
benchmark), one thing that's apparent is that we have quite a few parameters
here and that tuning them is likely to be a challenge.
One parameter in particular is the batch "objective", which controls how many
requests we want to see in the last batching interval before we choose to
batch (this is `rendezvous_dispatch_min_threshold`).
The problem with this is this is that there is no good, real-world, metric to
set it based on. This in contrast to the other parameters we have, which do
have some reasonable metric to compare to:
- rendezvous_dispatch_delay_ms: this is overhead we add to queries, so it
should be small & on the order of query execution latency (i.e. a few ms).
- rendezvous_dispatch_max_threshold: this controls how big our batches get, so
it should be on the order of what makes a SQL query too big (i.e. less than
a hundred records).
In contrast, we want to set `rendezvous_dispatch_min_threshold` such that
batching kicks in before we start using too many concurrent connections (which
is what query batching seeks to reduce), but the problem is that those two
numbers aren't directly connected. One clear problem, for example, is that if
our DB is in-region vs. out of-region, then for a given query execution time,
and a desired concurrency level before batching kicks in, we'd need different
values of `rendezvous_dispatch_min_threshold` (it would have to kick in faster
for the out-of-region workload).
So, this diff updates rendez vou to actually track concurrent connection count
before we force batching. This is the actual metric we care about here, and it
has a pretty natural "real world" values we can look at to decide where to set
it (our connection pool — which is limited at 100 concurrent connections —, and
our open connection baseline).
Note: I set this at 5 because that's more or less what servers look like
outside of spikes for Bonsai hg mapping, and of Changesets where I'm planning to
introduce this in the future:
- bonsai: https://fburl.com/ods/6d4a9qb5
- changesets: https://fburl.com/ods/kuq5x1vw (note: to make sense of this,
focus on just one server, otherwise the constnat spikes we get sort of hide
the big picture).
Reviewed By: farnz
Differential Revision: D27792603
fbshipit-source-id: 1a9189f6b50d48444b3373bd1cb14dc51b85a6d2
Summary:
Like it says in the title. There's no reason for this to be ad ad-hoc "throw in
an arg" when everything else is done by adding arg types.
Reviewed By: HarveyHunt
Differential Revision: D27791333
fbshipit-source-id: 38e5a479800179b249ace5cc599340cb84eb53e2
