Summary:
Some linkers (lld being one) use fallocate() or posix_fallocate() on
the output file before writing its contents. EdenFS would return
ENOSYS or ENOTSUP so glibc would fall back and write a single byte to
every 512 byte block, which is terribly slow and generates a bunch of
fake traffic in the Watchman journal.
This diff implements basic support for FUSE_FALLOCATE, avoiding this
slow emulation.
Reviewed By: xavierd
Differential Revision: D25934694
fbshipit-source-id: c6c90ea2b517d4dbedce29d9a4340870c8c177c3
Summary:
A lot of users have been having trouble getting their eden repos recloned
recently, to make this process simplier I add the reclone process to the clone
script.
In this script I check if a user has multiple repos depending on the same
backing store. This allows me to warn them that they might lose changes from
these other repos. This diff threads along the backing store to the
`eden list` result for that check.
Reviewed By: chadaustin
Differential Revision: D25078423
fbshipit-source-id: 9ceb1f9acc4ec170cbb12d4b0b3b7d51987f88e3
Summary:
The both options have basically the same value.
This is my next step for resolving mismatches between CC dynamic config and the current configuration and generally improving CC configuration.
Reviewed By: DurhamG
Differential Revision: D25973556
fbshipit-source-id: aae21efcd5174ed58efcb9e5d8c85831d35777ea
Summary:
In D25954401 (75e60ef4db) I by mistake removed a big chunk of an integration test. This diff
adds it back.
Reviewed By: krallin
Differential Revision: D25969563
fbshipit-source-id: 7b4143359e99a1ad0182d0561b6c0ab6af4295e0
Summary:
I'd like to eventually add an actual HTTP stack in Mononoke Server. Doing this
will be easier / nicer if we don't have to peek at the traffic to decide how
to interpret it (i.e. ALPN vs. HTTP).
To do so, we can use ALPN. If the client tells us they want the hgcli protocol,
we can use that, and skip peeking at the traffic entirely.
Obviously, we can't roll this out in one go, so first, let's accept traffic
advertised as hgcli via ALPN. Later, we can stop peeking at the traffic
entirely once we're not receiving any traffic from hgcli not advertised via
ALPN. I added ODS counters so we can canary this.
Note that the way I set this up is that if the client requests something that
isn't ALPN (e.g. H2), we just continue the handshake but don't select anything.
Finally, note that client side, we don't care (or even try to read) what the
server actually selected.
Reviewed By: johansglock
Differential Revision: D25954535
fbshipit-source-id: 183f6f56b2c8895aa8b18101565a4f2cd643be8d
Summary:
I'd like to update our connection acceptor to support ALPN in order to
distinguish hgcli traffic from http traffic (the former will have ALPN and the
latter will not).
To do so, I need to parse ALPN, which is a length-prefixed bytestring format.
Let's add support for it in a library. This is what this does.
Reviewed By: johansglock
Differential Revision: D25954536
fbshipit-source-id: 3756dde90906a5f0f36630b866bcb17d03bc94c6
Summary:
Make local build is broken with
```
building 'hgmain' binary
error: failed to select a version for the requirement `indicatif = "^0.14"`
candidate versions found which didn't match: 0.15.0
location searched: directory source `/data/users/liubovd/fbsource/fbcode/eden/scm/../../../third-party/rust/vendor` (which is replacing registry `https://github.com/rust-lang/crates.io-index`)
required by package `hgcommands v0.1.0 (/data/users/liubovd/fbsource/fbcode/eden/scm/lib/hgcommands)`
... which is depended on by `hgmain v0.1.0 (/data/users/liubovd/fbsource/fbcode/eden/scm/exec/hgmain)`
perhaps a crate was updated and forgotten to be re-vendored?
error: compilation of Rust target 'hgmain' failed
make: *** [Makefile:68: local] Error 1
```
Reviewed By: krallin
Differential Revision: D25973253
fbshipit-source-id: 42c66dcfe26dc5c7713d4bdedc2b353aa5eb20ed
Summary:
Scrub was being enabled by mutating the BlobConfig into the BlobConfig::Scrub.
This change removes the BlobConfig::Scrub variant as its not present in the thrift config any more, so there is not need for it to exist. Scrubbing is an optional mode of multiplexed store construction, so instead add ScrubOptions to BlobstoreOptions.
Reviewed By: StanislavGlebik
Differential Revision: D25927253
fbshipit-source-id: 29fceae59be8f7068300a7b8b298c6edbe4da04b
Summary: Add a ScrubOptions struct for the scrub blobstore in preparation for passing it more arguments in the next diff.
Reviewed By: StanislavGlebik
Differential Revision: D25927255
fbshipit-source-id: 5990ecd1b7ebf19b518306122045f670ad75fb0d
Summary:
Updating to a newer version of Tokio breaks hgcli.
This seems to be happening because Tokio fixed a bug where shutting down your
runtime could leak threads (https://github.com/tokio-rs/tokio/pull/2649), which
results in the runtime refusing to shutdown if you have a thread waiting to
read on stdin (note: I didn't try to back that out specifically to confirm).
Unfortunately, we have that in hgcli, because we usually kick off a stdin read
before we know if we'll need the data or not.
Here's the backtrace for a blocked shutdown P163996251.
We were already trying to shut down the runtime and ignore every outstanding
work (because we ensure earlier that there is nothing outstanding), so let's
just try even harder.
Reviewed By: StanislavGlebik
Differential Revision: D25952430
fbshipit-source-id: ae3a1413790cf81a5d990220f2dc5d4599219f73
Summary:
There are a few reasons why debugging with `--debug` is not optimal right now:
- It's in a namespace, so you can't `gdb -p PID`
- We use chg, so finding hg processes is tricky
This fixes that.
Note that running with `--debug` is very different from what we do in normal
mode anyway so there isn't much sense in keeping them super similar.
Reviewed By: StanislavGlebik
Differential Revision: D25952429
fbshipit-source-id: d15921dd008dd75de8c5f44e06b331f6c55adf58
Summary:
We had a user whose `hg sl` was completely hosed because they didn't have
master anymore (or hardly any remote bookmarks for that matter), which made
everything draft.
Let's make this easier to debug by logging subscriptions to hg rage.
Reviewed By: StanislavGlebik
Differential Revision: D25951745
fbshipit-source-id: d08b6bf7febe47923908b312f1881e3be15188c8
Summary: Adding protobuf manifest, which will be used to compile f4d/velox.
Reviewed By: spershin
Differential Revision: D25926384
fbshipit-source-id: 08a2dffc9e9baa0d393e5f648ce99ca032a9df5f
Summary: Adding lzo manifest, which will be used by f4d/velox
Reviewed By: amitkdutta
Differential Revision: D25926434
fbshipit-source-id: 4d7efe5eba7cdca1f682fbd4cfd6ff0337ce7c19
Summary:
I gathered information from various sources in this newdoc entry.
The goal is to have an easy link that gives a general summary of what
Segmented Changelog is and what it does.
Reviewed By: quark-zju
Differential Revision: D25908322
fbshipit-source-id: 8f3d2725afe8db5a9bc179189c287fd54aaca08b
Summary:
In previous diffs we added a new way to change xrepo mapping - just landing a
commit with a special extra is enough to change the mapping version used to
rewrite a commit. This diff adds an admin command that let's us do this.
Reviewed By: ikostia
Differential Revision: D25954401
fbshipit-source-id: 8ff3366dc305a6fea8bb013e457ad47fa7163994
Summary:
This is implements http keep-alive behavior for `/netspeedtest` endpoint. `keep-alive` is implicit for `HTTP/1.1`. It means that connection is not closed after sever sends a response. Further request/response pairs can be exchanged.
This enables warmup measurements in netspeedtest before real measurements are made.
The difference is quite significant, even when I am measuring this server-side.
Reviewed By: krallin
Differential Revision: D25946675
fbshipit-source-id: a7b0f812bf61d704ca93615ab09d0897a08f1ba1
Summary:
Now, when direct connection to mononoke is being rolled out, hg_speed_test (part of hg debugnetwork) doesn't work anymore. It was a feature of hg_ssh_wrapper.
This adds server-side of hg_speed_test using HTTP.
Reviewed By: HarveyHunt
Differential Revision: D25883922
fbshipit-source-id: 1522b3db69ff36899bd45743374e59668bda4048
Summary:
In the previous diff we added a special commit extra which allows backsyncer to
change the mapping version that was used to rewrite commits between repos.
This is a dangerous thing since it might corrupt the repository. So to avoid
people accidentally corrupting the repository let's add a check that would
prevent landing public commits with change-xrepo-mapping extra.
Reviewed By: ahornby
Differential Revision: D25951529
fbshipit-source-id: d2de37d1bf2f2eb9f45b3bf3e21706a799a50f69
Summary:
At the moment the procedure to change commit sync mapping version is fairly
complicated [1]. This diff attempts to simplify one aspect of it.
At the moment we need to disable pushredirection and manually sync a commit
between two repos. Disabling pushredirection is quite error prone and led to
SEVs in the past. With this diff we just need to create a commit with a special
extra "change-xrepo-mapping" and let the backsyncer rewrite a commit with the
new version for us.
This means that landing a commit with this extra should not be allowed to
everyone - I'm going to change bookmark movement to check that.
Reviewed By: ikostia
Differential Revision: D25950531
fbshipit-source-id: 78b1f4950e6bb7da8b511e02685ed6084c29a680
Summary:
This implements chunking of the original iterator by saturation of some
accumulator until a condition starts to be satisfied.
Note: I tried looking through `Vec`, `itertools` and `Iterator` methods, and
did not find anything that would allow me to express this easily.
Reviewed By: StanislavGlebik
Differential Revision: D25947821
fbshipit-source-id: 9e4dd738ecd2ab06ebb69123e4a03059f96b3fb6
Summary:
This option limits the total number of files copied/removed. I will also add a
mode to chunk copying/removal into multiple commits, let's rename this, so that
it's not confusing.
Reviewed By: ahornby
Differential Revision: D25947233
fbshipit-source-id: 204ef3c0e610072c87548b7534f575fb6fe21fc6
Summary:
This change makes it possible to diff SmallRepoCommitSyncConfigs.
This can be useful to find what's changed between two mapping versions, and it
will be used in the later diffs as well.
Reviewed By: ikostia
Differential Revision: D25947041
fbshipit-source-id: 96b0447e78278c826998ba236bc27bb65daf0fef
Summary:
The exit condition for looking for the `.arcconfig` file will never be reached
on Windows, as the root directory is usually something like `C:\\`.
Instead, stop if we get to a point where `os.path.dirname` returns the same
thing, as that will probably also be a root.
Reviewed By: singhsrb
Differential Revision: D25947124
fbshipit-source-id: 30f4bf3df0b4119694194bcb1c8bcadca41556e4
Summary:
This feature is useful for testing time-dependent stuff (e.g. it
allows you to stop/forward time). It's already included in the buck build.
Reviewed By: SkyterX
Differential Revision: D25946732
fbshipit-source-id: 5e7b69967a45e6deaddaac34ba78b42d2f2ad90e
Summary:
quark-zju made it much safer to rollout configs recently by enforcing
time-sharding, but one little downside is that it becomes harder to test
them locally (or at diff-time, which I'm working on adding) because the time
shard will usually not pass.
This diffs add an environment variable to force all shards to pass.
Reviewed By: DurhamG
Differential Revision: D25925207
fbshipit-source-id: 343b90165ad2b6bae64b6821eae95c58f7d79698
Summary: We will use it in a separate subcmd later.
Reviewed By: StanislavGlebik
Differential Revision: D25944075
fbshipit-source-id: 211063ef74b192c18b5d006e9e0592085fdfc811
Summary: This enum no longer serves a purpose, delete it
Reviewed By: ikostia
Differential Revision: D25927254
fbshipit-source-id: 6731d6a57313946ee70b301ebc4f5cd634a90e61
Summary:
I've been seeing this hundreds of times / day in my Eden logs:
```
2021-01-07 09:19:19,702 Fetching failed, continuing as this may be spurious: Expected type that converts to PySequence but received set
```
Unfortunately, those logs are really not very helpful, because that error
doesn't tell you anything about where it occurred. I do however care about them
because they have correlated with slowness in Eden for me :)
At the same time, this error is a bit important: The last time we had an issue
here, it turned into a SEV. That was because the fallback was buggy, but it
does suggest that we do not expect to be routinely falling back.
So, let's log more.
Reviewed By: xavierd
Differential Revision: D25924077
fbshipit-source-id: f59d81cc51786f94f747142ec4baaca78f3072c2
Summary:
To ensure that a specific content gets sharded to multiple tasks (by tasks_per_content config parameter) each content can have multiple routing keys. These are the SHA256 (as before) or the SHA256 suffixed with "-1", "-2" etc..
When the routing key is generated for an item, one of them is picked randomly.
Added config constraint to ensure that the number of tasks parameter is at least 1.
Reviewed By: krallin
Differential Revision: D25886402
fbshipit-source-id: fb8911dad07d2f0b6bbf57b4ede084428fe6c49d
