Summary:
In order to get Watchman to recognize EdenFS mounted as NFS, we previously set
the f_mntfromname to be "edenfs". Unfortunately, Apple decided that when the
NFS server is accessed via a unix socket it would overwrite that and instead
use the path to the socket.
Digging in the source code, I did find that the f_fstypename can be overwritten
by calling fsctl with the right set of arguments. Thus, let's do just that.
Reviewed By: kmancini
Differential Revision: D28270605
fbshipit-source-id: f6be4e394d814806aa03ec3e82b8bc2faf364ea7
Summary:
Due to NFS being designed as a network filesystem, it default to binding on a
TCP socket. For EdenFS, since we're not expecting to mount an actual remote
filesystem, we bind these sockets to localhost. Unfortunately, TCP sockets have
some inherent overhead due to being designed to be reliable over a non-reliable
medium.
On macOS, Apple provides a way to mount an NFS server that is listening on a
unix domain socket. Thanks for unix socket being reliable, the TCP overhead
goes away leading to some higher throughput and lower latency for the NFS
server. For EdenFS, timing `rg foobar` over a directory containing 27k files gives:
NFS over TCP:
rg foobar > /dev/null 0.80s user 5.44s system 567% cpu 1.100 total
NFS over UDS:
rg foobar > /dev/null 0.77s user 5.27s system 679% cpu 0.888 total
osxfuse:
rg foobar > /dev/null 0.87s user 5.59s system 662% cpu 0.975 total
The main drawback of going through a unix socket is that D27717945 (bcf6aa465c) appears to
no longer be effective due to
8f02f2a044/bsd/nfs/nfs_vfsops.c (L3739)
Reviewed By: kmancini
Differential Revision: D28261422
fbshipit-source-id: 25dc1dc78cdb50d6c6550a86ef01ea2c894c110f
Summary:
macOS supports NFS servers that can be reached via a unix socket as a way to
improve performance by reducing the TCP cost. To support this, let's first
allow the socket to bind to to be passed to the RpcServer, and then pass it
through to the privhelper code.
Reviewed By: kmancini
Differential Revision: D28261423
fbshipit-source-id: 78c60aac26353d1da76a67897429b964332df8b3
Summary: Looks like this is causing files to not be immediately updated after an update.
Reviewed By: kmancini
Differential Revision: D28384151
fbshipit-source-id: 61159db64a9f3c9c1f9e54ee0462ca870ff2aecc
Summary:
This struct is intended to be a single entry-point to the megarepo logic. It is also intended to be owned directly by scs_server without the `Mononoke` struct (from `mononoke_api`) intermediary. In effect, this means that mononoke server won't be affected by `MegarepoApi` at all.
Apart from adding this struct, this diff also adds instantiation of prod `AsyncMethodRequestQueue` and wires it up to the scs_server to enqueue and poll requests.
Reviewed By: StanislavGlebik
Differential Revision: D28356563
fbshipit-source-id: b67ee48387d794fd333d106d3ffd40124086c97e
Summary: I use tags extensively and I love them to be supported as well.
Reviewed By: asm89
Differential Revision: D28348565
fbshipit-source-id: 7d94d048b734c91e7d74a1c3efeefc87943066ad
Summary:
With `MegarepoApi` struct in play, there is a genuine need to have two repo
factories in a single process: this allows the structure to be self-sufficient
and instantiated without any references to `Mononoke` from `monooke_api`.
While this need could be solved by just wrapping a `RepoFactory` in an `Arc`,
it seems like most of it is already clonable, so let's just make it fully
clonable by fixing a few remaining places. (I prefer this over `Arc`, because
there's less refactoring of unrelated code). Given that there will likely be a
single digit of repo factories instantiated in a single process, the difference
between a single arc's clone (`Arc<RepoFactory>`) and ~10 arc clones (what I
did) is negligible.
Differential Revision: D28379860
fbshipit-source-id: fbddbdc913fedcd5846366344bc2f2c1ec4bd91e
Summary: This way implementing MegarepoApi is more convenient.
Reviewed By: krallin
Differential Revision: D28355487
fbshipit-source-id: e7643e854ee46fe6cb9c4a882f6c677bf4e77262
Summary:
Partial getbundle optimization didn't work correctly if one merge parent was an ancestor
of another - it might return a parent commit before a child commit. Say we had a graph like this
```
C
| \
| B
| /
A
```
Previously partial getbundle optimization could have visited A before it
visited B, and returned commits in the wrong order, which in turn would lead to
"hg pull" failures. The reason for that is that we don't order commits by generation number.
This diff fixes it by using UniqueHeap to sort commits by generation number
before returning them. Also note that as the name suggests UniqueHeap stores
only unique values in the heap, so we don't need to keep separate `visited`
hashset
Reviewed By: krallin
Differential Revision: D28378145
fbshipit-source-id: 9467fb7cfa8386e9e2725b80f386240b522ff3ee
Summary:
Just as with rewrite_commit function that I moved a to commit_transformation
not so long ago (D28259214 (df340221a0)), let's also move copy_file_contents. The motivation
is because we are going to use it in the next diff for sync_changeset method.
Reviewed By: ikostia
Differential Revision: D28352402
fbshipit-source-id: 12288a51540c9793d988e4063735bcbc1c3b1a7f
Summary:
Those blobs are designed to fit in cachelib, so we shouldn't attempt to
zstd-encoded them (not to mention that they don't usually compress very well
since many of those blobs come from binary files, though that's not true of
all).
However, we weren't actually doing this right now. There were a few reasons:
- Our threshold didn't allow enough headroom. I don't know when exactly this
got introduced (of indeed if that has worked since we introduced cachelib
compression).
- We serialize a bunch of extra metadata that we really don't need as it's a
bit meaningless once it's gone through the cache (we notably don't serialize
this in Memcache). This diff updates us to just store bytes here.
Differential Revision: D28350387
fbshipit-source-id: 4d684ab58cea137044e20951ec4cbb21240b8dfc
Summary: See the next diff for motivation: this makes it easier to implement.
Differential Revision: D28350388
fbshipit-source-id: 026605cf8296a945d6cc81b7f36d9198325bf13c
Summary:
Test is flaky: https://www.internalfb.com/intern/test/281474993296146?ref_report_id=0
I suppose this happens due to mechanics how we measure qps - with low volume, some averaging or bucketing might not work as precisely as with a lot of qps we have in normal, prod scenarios.
Lowering the threshold by 1 should fix this.
Reviewed By: ahornby
Differential Revision: D28350150
fbshipit-source-id: 694bfb8cce1935704e35b27f7d4455439d4bfffe
Summary:
I should've made them structs from the beginning, but of course I thought that
I know better and these tokens can definitely not be richer than just strings.
Well, it turns out we need them to be richer. Specific reason is that in theory
a single Mononoke (or scs_server) instance can run with multiple storage
configs. For us this means that one target's requests may be stored in one
db, while another target's requests - in another one. For blobstores this is
even used in practice, while for xdb it's just a theoretical thing, but we need
to support it nevertheless.
To do so, let's add the ability to query the target (and, correspondingly, the
Mononoke repo) from any king of params our async methods receive: ThriftParams
or Token implementors.
In addition, this diff really implements `AddScubaParams` and `AddScubaResponse` for more things than before, so there's that.
Finally, apart from making tokens structured, this diff also changes an interface in two more ways:
- it adds optional `message` fields to several params structs
- it adds `changesets_to_merge` to `MegarepoChangeTargetConfigParams`
Reviewed By: StanislavGlebik
Differential Revision: D28333999
fbshipit-source-id: 99bd19b040b59ee788ef661dda3171cc56254d33
Summary: Instead of passing a client certificate path to libcurl, load the certificate into memory and pass it to libcurl as a blob using `CURLOPT_SSLCERT_BLOB`. This allows us to convert the certificate format in-memory from PEM to PKCS#12, the latter of which is supported by the TLS engines on all platform (and notably SChannel on Windows, which does not support PEM certificate).
Reviewed By: quark-zju
Differential Revision: D27637069
fbshipit-source-id: f7f8eaafcd1498fabf2ee91c172e896a97ceba7e
Summary:
The Rust `openssl` crate will using dynamic linking by default when built with `cargo`. This is a problem on Windows, since we only support cargo-based builds on that platform, but OpenSSL is not present in the system's shared library search paths.
Since we already have a copy of OpenSSL uploaded to LFS, the simplest solution is to just copy the required DLLs right next to the Mercurial executable so that they will be found at launch.
A better solution would probably be to use static linking here. From reading the crate's documentation (and build script), it seems like setting `OPENSSL_STATIC=1` during the build should force static linking, but in practice I have not been able to get this to work.
Reviewed By: DurhamG
Differential Revision: D28368579
fbshipit-source-id: 3fceaa8d081650d60356bc45ebee9c91ef474319
Summary:
split full sync into 3 steps
Commit cloud by default pulls only 30 days of commits.
Users often see some of their commits are missing in their smartlog.
I discovered that most of the users know the '--full' option (`hg cloud sync --full`) but not the 'max_sync_age' config option.
So, they try --full option but it could fail due to very very old commits we haven't migrated to Mononoke.
Users often don't really need those commits but it's not nice that the whole sync run failed.
We know that at least latest 2 years of commits are present in Mononoke.
So if we split a bit how we sync with --full option works, it would at least result in partially successfully sync for the latest 2/3 years of commits.
Reviewed By: mitrandir77
Differential Revision: D28352355
fbshipit-source-id: b5bacd7d5256191528613e3c0bcbb21b0104ac3c
Summary:
deprecate 4 commits at a time limitation for unhydrated pulls
This could speedify cloud join commands significantly (by many X times) and hg cloud sync --full command.
Reviewed By: farnz
Differential Revision: D28351849
fbshipit-source-id: f9f3d7a5c07d61cb51a5bb6284afaad963662c94
Summary: This is going to be used in the next diffs.
Reviewed By: StanislavGlebik
Differential Revision: D28333977
fbshipit-source-id: ad52d307e13ae9bd662209ef7ec6afdcf0ee24c7
Summary:
While ImmediateFuture are expected to be used on values that are mostly
immediate, there are cases where it won't be. In these cases we need a way to
wait for the computation/IO to complete. In order to achieve this, we need to
transform an ImmediateFuture onto a SemiFuture.
Reviewed By: fanzeyi
Differential Revision: D28293941
fbshipit-source-id: 227c0acf1e22e4f23a948ca03f2c92ccc160c862
Summary:
When a T can be default constructed, make an ImmediateFuture default
constructible.
Reviewed By: fanzeyi
Differential Revision: D28292874
fbshipit-source-id: 4c239cc9c3f448652b2bcdc103ea1a81ace46402
Summary: This should help the compiler generate even better code.
Reviewed By: chadaustin
Differential Revision: D28153979
fbshipit-source-id: b1d84c92af4fa760c92624c53d7f57330d7706fa
Summary:
This has been here for a little while, but it's worth changing. Currently, we
entirely discard logs coming via a CoreContext in EdenAPI.
We don't typically log many of those anywhere in Mononoke, but when we do they
tend to be for error conditions, warnings, or aggregated reporting, and can be
quite meaningful as a result.
So, let's update to not discard them. To make it easy to differentiate those
logs from EdenAPI request-level logs (even though they'll both have `service` =
`edenapi`), let's give the latter a Log Tag (which is consistent with what
we do in repo client).
Differential Revision: D28350733
fbshipit-source-id: 3b12a4b56f28435460186e1f7578163ca7bdaebc
