Summary:
Add a counter to report the number of mounts that we failed to remount during
startup. Mount failures do not prevent EdenFS startup from proceeding. It is
useful to have a metric to report if these errors did occur even though the
start-up as a whole still proceeded otherwise.
Reviewed By: chadaustin
Differential Revision: D20319512
fbshipit-source-id: fd503a1ccc91b476cc9dc2bc6323501bbbeaf2c5
Summary:
On Python 2 the Mercurial JSON can be binary, which can break the telemetry
logger. Ensure the JSON string is valid utf-8 (although it might still be
invalid JSON).
Reviewed By: xavierd
Differential Revision: D20343845
fbshipit-source-id: 61e99e742bddf23c7fd5354a5754d79a0a452c28
Summary:
The goal of the whole stack is quite simple (add reponame field to BlobRepo), but
this stack also tries to make it easier to initialize BlobRepo.
To do that BlobrepoBuilder was added. It now accepts RepoConfig instead of 6
different fields from RepoConfig - that makes it easier to pass a field from
config into BlobRepo. It also allows to customize BlobRepo. Currently it's used
just to add redaction override, but later we can extend it for other use cases
as well, with the hope that we'll be able to remove a bunch of repo-creation
functions from cmdlib.
Because of BlobrepoBuilder we no longer need open_blobrepo function. Later we
might consider removing open_blobrepo_given_datasources as well.
Note that this diff *adds* a few new clones. I don't consider it being a big
problem, though I'm curious to hear your thoughts folks.
Note that another option for the implementation would be to take a reference to objects
instead of taking them by value. I briefly looked into how they used, and lot of them are passed to the
objects that actually take ownership of what's inside these config fields. I.e. Blobstore essentially takes ownership
of BlobstoreOptions, because it needs to store manifold bucket name.
Same for scuba_censored_table, filestore_params, bookmarks_cache_ttl etc. So unless I'm missing anything, we can
either pass them as reference and then we'll have to copy them, or we can
just pass a value from BlobrepoBuilder directly.
Reviewed By: krallin
Differential Revision: D20312567
fbshipit-source-id: 14634f5e14f103b110482557254f084da1c725e1
Summary:
We've recently added new scuba table for derived data
(https://fburl.com/scuba/mononoke_derived_data/e4sekisf), and looks like our
previous format of logging is not very useful. It's better to have separate
fields for changeset id and derived data type, since it makes aggregation
easier.
Reviewed By: krallin
Differential Revision: D20309093
fbshipit-source-id: 48f5f04e0412002ef04028e34b12bf267a9b6834
Summary:
Spliting lock file acquisition from `IdDag::prepare_filesystem_sync` to its own
function.
Useful when looking ahead to split IdDag from IndexedLog.
Reviewed By: quark-zju
Differential Revision: D20316443
fbshipit-source-id: a0fd43439730376920706bb4349ce497f6624335
Summary:
This removes an inline use of the indexedlog indexes.
This is going to be useful when we try to separate IndexedLog specifics from
IdDag functionality.
Reviewed By: quark-zju
Differential Revision: D20316058
fbshipit-source-id: 942a0a71660bb327376c81fd3ac435d002ecca6e
Summary:
Add `hg debugnetwork` to the things that `hg rage` collects.
Some of the output from `hg debugnetwork` comes from the peer connection, which is not captured when using `ui.pushbuffer`, as it is written using the remote ui object created when the peer connection is set up.
To handle this, add `ui._outputui` as a way to redirect ui output to another ui object, and use this to redirect the output from the remoteui to the local ui object which is buffering output for hg rage.
Reviewed By: quark-zju
Differential Revision: D20307725
fbshipit-source-id: 3b79a77a39c6e2c5f8a7d5cc271ec466653d4db3
Summary:
The mock crates contain a standard set of mock commits ids with all nybbles
set to a single hex value.
For the mutation tests I want to be able to generate them from a number and
have more than 15 changeset IDs. Add a new function that generates an hg
changeset ID from a number.
Reviewed By: krallin
Differential Revision: D20287382
fbshipit-source-id: 1f57de89f19e2e2eea8dbfea969a4d54510e23d8
Summary:
Note that comparing to many other asyncifying efforts, this one actually adds
one more clone instead of removing them. This is the clone of a logger field.
That shouldn't matter much because it can be cleaned up later and because this
function will be called once per repo.
Reviewed By: krallin
Differential Revision: D20311122
fbshipit-source-id: ace2a108790b1423f8525d08bdea9dc3a2e3c37c
Summary:
When checking to make sure the destination directory does not look mounted,
ignore ENOTCONN errors. This can happen if the directory is from a previous
EdenFS instance and did not get unmounted from the kernel properly.
While ideally it would be nicer to run `umount -lf` to clean up this stale
mount point, simply mounting a new checkout on top of this directory is fine.
We don't try to clean up the old mount point here since `umount -lf` would
require root priviliges.
Reviewed By: wez
Differential Revision: D20313117
fbshipit-source-id: c25c85d6b80750662b301f0dff8f293299770a4d
Summary:
Add a command that performs various network tests to check the connection to
the Mercurial server for SSH peers.
These tests are:
* Check we can look up the server name in DNS.
* Check we can make a TCP connection to the SSH server.
* Check we can connect and authenticate over SSH and run `hostname`.
* Check we can make a Mercurial wireproto connection and find the `master` bookmark.
* Check the download and upload speed to the server.
Checking the speed requires a companion speed test program on the server.
Reviewed By: mitrandir77, farnz
Differential Revision: D20305227
fbshipit-source-id: adba02a6a1c40ffc20d7bf9d962a5fcf85e06544
Summary:
Instead of returning `anyhow::Error` wrapping an `ErrorKind` enum
from each Thrift client method, just return an error type specific
to that method. This will make error handling simpler and less
error-prone by removing the need to downcast the returned error.
This diff also removes the `ErrorKind` enums so that we can be sure
that there are no leftover places trying to downcast to them.
(Note: this ignores all push blocking failures!)
Reviewed By: dtolnay
Differential Revision: D20260398
fbshipit-source-id: f0dd96a7b83dd49f6b30948660456539012f82e6
Summary: Pass a valid ObjectFetchContext down into certain untracked requests.
Reviewed By: chadaustin
Differential Revision: D20243575
fbshipit-source-id: e7112c3bab1265803a26130c4d72905c25f2e729
Summary: Make a bytestring with 'b""' to fix Python3
Reviewed By: DurhamG
Differential Revision: D20287313
fbshipit-source-id: 7455d1509684bfb0857a3b060bdcca7e658343fd
Summary: We need to encode/decode utf8 when reading/writing to the hgrc file.
Reviewed By: DurhamG
Differential Revision: D20286068
fbshipit-source-id: b1d6828fb62a83ad22414de6883004411f302142
Summary:
Previously we could have "Started ..." before "Starting ..."
This diff fixes it.
Reviewed By: krallin
Differential Revision: D20277406
fbshipit-source-id: 3c2f3fa1723c2e0852c6b114592ab7ad90be17ff
Summary:
We saw >10 timeouts on Mononoke side from people who fetch a lot of files from corp network (https://fburl.com/scuba/mononoke_test_perf/qd15c5f1). They all tried to download a lot of files, and they 90 mins timeout limit.
Let's try to download these files in rather large chunks (e.g. 200000 files). That should help with resumability and prevent timeout errors, and make retries smaller.
It adds an overhead for establishing a new connection after every 200000 files. That shouldn't be a big problem, and even if it's a big problem we can just increase remotefilelog.prefetchchunksize.
Reviewed By: xavierd
Differential Revision: D20286499
fbshipit-source-id: 316c2e07a0856731629447627ae65c094c6eecc5
Summary: expose the counters for number of pending imports (blobs, trees, prefetches) to allow use in tooling
Reviewed By: chadaustin
Differential Revision: D20269853
fbshipit-source-id: d2b7e2110520290751699c4a891d41ebd5b374cf
Summary:
On bad network link (such as on VPN), the reliability of the connection to
Mercurial might be fairly flaky. Instead of failing fetching files, let's retry
a bit first, in the hope that the connection will be back by then.
Reviewed By: quark-zju
Differential Revision: D20295255
fbshipit-source-id: d3038f5e4718b521ae4c3f2844f869a04ebb25e3
Summary:
The old code does "read, lock, write", which is unsound because after "lock"
the data just read can be outdated and needs a reload.
Reviewed By: xavierd
Differential Revision: D20306137
fbshipit-source-id: a1c29d5078b2d47ee95cf00db8c1fcbe3447cccf
Summary:
This updates the store_bytes method to chunk incoming data instead of uploading
it as-is. This is unfortunately a bit hacky (but so was the previous
implementation), since it means we have to hash the data before it has gone
through the Filestore's preparation.
That said, one of the invariants of the filestore is that chunk size shouldn't
affect the Content ID (and there is fairly extensive test coverage for this),
so, notionally, this does work.
Performance-wise, it does mean we are hashing the object twice. That actually
was the case before as well anyway (since obtain the ContentId for FileContents
would clone them then hash them).
The upshot of this change is that large files uploaded through unbundle will
actually be chunked (whereas before, they wouldn't be).
Long-term, we should try and delete this method, as it is quite unsavory to
begin with. But, for now, we don't really have a choice since our content
upload path does rely on its existence.
Reviewed By: StanislavGlebik
Differential Revision: D20281937
fbshipit-source-id: 78d584b2f9eea6996dd1d4acbbadc10c9049a408
Summary:
This is a very small struct (2 u64s) that really doesn't need to be passed by
reference. Might as well just pass it by value.
Differential Revision: D20281936
fbshipit-source-id: 2cc64c8ab6e99ee50b2e493eff61ea34d6eb54c1
Summary: separate out the Facebook-specific pieces of the sql_ext crate
Reviewed By: ahornby
Differential Revision: D20218219
fbshipit-source-id: e933c7402b31fcd5c4af78d5e70adafd67e91ecd
Summary:
We've had cases where a git commit goes in that shouldn't be translated
to Mercurial. Let's add an option to skip the commit. Instead of skipping it
entirely (which would require complicated logic to then parent the following
commit on the last converted commit), let's just convert the skipped commit as
an empty commit. This should cover the cases we've encountered so far.
Reviewed By: krallin
Differential Revision: D20261743
fbshipit-source-id: da401863b09c2ac727aae1ceef10a0e8d8f98a7e
Summary:
Context: https://fb.workplace.com/groups/rust.language/permalink/3338940432821215/
This codemod replaces all dependencies on `//common/rust/renamed:tokio-preview` with `fbsource//third-party/rust:tokio-preview` and their uses in Rust code from `tokio_preview::` to `tokio::`.
This does not introduce any collisions with `tokio::` meaning 0.1 tokio because D20235404 previously renamed all of those to `tokio_old::` in crates that depend on both 0.1 and 0.2 tokio.
This is the tokio version of what D20213432 did for futures.
Codemod performed by:
```
rg \
--files-with-matches \
--type-add buck:TARGETS \
--type buck \
--glob '!/experimental' \
--regexp '(_|\b)rust(_|\b)' \
| sed 's,TARGETS$,:,' \
| xargs \
-x \
buck query "labels(srcs, rdeps(%Ss, //common/rust/renamed:tokio-preview, 1))" \
| xargs sed -i 's,\btokio_preview::,tokio::,'
rg \
--files-with-matches \
--type-add buck:TARGETS \
--type buck \
--glob '!/experimental' \
--regexp '(_|\b)rust(_|\b)' \
| xargs sed -i 's,//common/rust/renamed:tokio-preview,fbsource//third-party/rust:tokio-preview,'
```
Reviewed By: k21
Differential Revision: D20236557
fbshipit-source-id: 15068b93a0a944d6249a1d9f63840a4c61c9c1ba
Summary:
I thought the index function could be the bottleneck. However, the Log reading
(xxhash, decoding vlqs) can be much slower for very long entries. Therefore
using bytes as the lag threshold is better. It does leaked the Log
implementation details (how it encodes an entry) to some extend, though.
Reverts D20042045 and D20043116 logically. The lagging calculation is using
the new Index::get_original_meta API, which is easier to verify correctness
(In fact, it seems the old code is wrong - it might skip Index flushes if
sync() is called multiple times without flushing).
This should mitigate an issue where a huge entry (generated by `hg trace`) in
blackbox does not get indexed in time and cause performance regressions.
Reviewed By: DurhamG
Differential Revision: D20286508
fbshipit-source-id: 7cd694b58b95537490047fb1834c16b30d102f18
