Summary:
Time filters for file history require fetching changeset or changeset info to decide whether to include the commit into response. To speed up the process instead of sequential mapping, I buffer the map stream in batches of 100.
However it is unfortunate to fetch additional 100 history changesets if there is no time filters and only several commits were requested. Keeping also in mind that the most of the requests don't care about time.
Avoiding that would speed up the history generation,
I changed the changeset_path API so now it just returns the stream of changeset contexts. And commit_path API for history if there is no time filters collects everything to vector, otherwise applies them and also collects to vector. Then this vector is converted into response using FuturesOrdered.
Reviewed By: StanislavGlebik
Differential Revision: D20287497
fbshipit-source-id: 0c6b1315eccddb48f67bf5fa732bdf7c9a54a489
Summary: "tags" are stored as sets, but must be serialized as lists in python. This only affects the `ExternalTelemetryLogger`, which is used on mac builds.
Reviewed By: simpkins
Differential Revision: D20351645
fbshipit-source-id: b097bd6aeb6d109323186f4ae4ef959f831ead2b
Summary:
A lot of callsites want to know repo name. Currently they need to pass it from
the place where repo was initialized, and that's quite awkward, and in some
places even impossible (i.e. in derived data, where I want to log reponame).
This diff adds reponame in BlobRepo
Reviewed By: krallin
Differential Revision: D20363065
fbshipit-source-id: 5e2eb611fb9d58f8f78638574fdcb32234e5ca0d
Summary:
We track this in Scuba right now (and alarm on it), but tracking it in ODS will
make it easier to incorporate in our canary and post-release health check
workflow.
Reviewed By: StanislavGlebik
Differential Revision: D20361803
fbshipit-source-id: 99fb514d41f9cda42c3c9a82f3b8d6681285430a
Summary:
Before we assumed that if the rows_affected length doesn't match the number of
entries we were trying to insert we have a conflict. Let's verify if we really
have conflict or we're trying to insert the same entry twice.
Reviewed By: krallin
Differential Revision: D20343219
fbshipit-source-id: 19e032439fdd65f5fe1afe1a10b401bc2fe33462
Summary: Running blobimport twice on the same commit seems to cause problems.
Reviewed By: krallin
Differential Revision: D20343218
fbshipit-source-id: 4d572630e7c15c219bee8db15cc879b2cb8602fe
Summary:
It might be useful to know how long the end-to-end wireproto command to find
the master bookmark took, so log that.
Reviewed By: quark-zju
Differential Revision: D20344686
fbshipit-source-id: 8459a378a0c828a929eee5f9ffd34875a5b8d3cc
Summary:
For high latency connections we will need to warm up the connection before
starting the download or upload test. Facilitate this by making it possible to
run multiple tests within the same connection.
This changes the protocol for the speed-test command to be line-oriented commands.
Reviewed By: farnz
Differential Revision: D20344687
fbshipit-source-id: 1d2a815736caf974ed1aaaaf365b86343d41fe02
Summary: Add ability to walk all published bookmarks as there may be multiple important bookmarks
Reviewed By: krallin
Differential Revision: D20249806
fbshipit-source-id: aff2ee1ec7d51a9e4fb6e1e803612abd207fd6cb
Summary:
Add a counter to report the number of mounts that we failed to remount during
startup. Mount failures do not prevent EdenFS startup from proceeding. It is
useful to have a metric to report if these errors did occur even though the
start-up as a whole still proceeded otherwise.
Reviewed By: chadaustin
Differential Revision: D20319512
fbshipit-source-id: fd503a1ccc91b476cc9dc2bc6323501bbbeaf2c5
Summary:
On Python 2 the Mercurial JSON can be binary, which can break the telemetry
logger. Ensure the JSON string is valid utf-8 (although it might still be
invalid JSON).
Reviewed By: xavierd
Differential Revision: D20343845
fbshipit-source-id: 61e99e742bddf23c7fd5354a5754d79a0a452c28
Summary:
The goal of the whole stack is quite simple (add reponame field to BlobRepo), but
this stack also tries to make it easier to initialize BlobRepo.
To do that BlobrepoBuilder was added. It now accepts RepoConfig instead of 6
different fields from RepoConfig - that makes it easier to pass a field from
config into BlobRepo. It also allows to customize BlobRepo. Currently it's used
just to add redaction override, but later we can extend it for other use cases
as well, with the hope that we'll be able to remove a bunch of repo-creation
functions from cmdlib.
Because of BlobrepoBuilder we no longer need open_blobrepo function. Later we
might consider removing open_blobrepo_given_datasources as well.
Note that this diff *adds* a few new clones. I don't consider it being a big
problem, though I'm curious to hear your thoughts folks.
Note that another option for the implementation would be to take a reference to objects
instead of taking them by value. I briefly looked into how they used, and lot of them are passed to the
objects that actually take ownership of what's inside these config fields. I.e. Blobstore essentially takes ownership
of BlobstoreOptions, because it needs to store manifold bucket name.
Same for scuba_censored_table, filestore_params, bookmarks_cache_ttl etc. So unless I'm missing anything, we can
either pass them as reference and then we'll have to copy them, or we can
just pass a value from BlobrepoBuilder directly.
Reviewed By: krallin
Differential Revision: D20312567
fbshipit-source-id: 14634f5e14f103b110482557254f084da1c725e1
Summary:
We've recently added new scuba table for derived data
(https://fburl.com/scuba/mononoke_derived_data/e4sekisf), and looks like our
previous format of logging is not very useful. It's better to have separate
fields for changeset id and derived data type, since it makes aggregation
easier.
Reviewed By: krallin
Differential Revision: D20309093
fbshipit-source-id: 48f5f04e0412002ef04028e34b12bf267a9b6834
Summary:
Spliting lock file acquisition from `IdDag::prepare_filesystem_sync` to its own
function.
Useful when looking ahead to split IdDag from IndexedLog.
Reviewed By: quark-zju
Differential Revision: D20316443
fbshipit-source-id: a0fd43439730376920706bb4349ce497f6624335
Summary:
This removes an inline use of the indexedlog indexes.
This is going to be useful when we try to separate IndexedLog specifics from
IdDag functionality.
Reviewed By: quark-zju
Differential Revision: D20316058
fbshipit-source-id: 942a0a71660bb327376c81fd3ac435d002ecca6e
Summary:
Add `hg debugnetwork` to the things that `hg rage` collects.
Some of the output from `hg debugnetwork` comes from the peer connection, which is not captured when using `ui.pushbuffer`, as it is written using the remote ui object created when the peer connection is set up.
To handle this, add `ui._outputui` as a way to redirect ui output to another ui object, and use this to redirect the output from the remoteui to the local ui object which is buffering output for hg rage.
Reviewed By: quark-zju
Differential Revision: D20307725
fbshipit-source-id: 3b79a77a39c6e2c5f8a7d5cc271ec466653d4db3
Summary:
The mock crates contain a standard set of mock commits ids with all nybbles
set to a single hex value.
For the mutation tests I want to be able to generate them from a number and
have more than 15 changeset IDs. Add a new function that generates an hg
changeset ID from a number.
Reviewed By: krallin
Differential Revision: D20287382
fbshipit-source-id: 1f57de89f19e2e2eea8dbfea969a4d54510e23d8
Summary:
Note that comparing to many other asyncifying efforts, this one actually adds
one more clone instead of removing them. This is the clone of a logger field.
That shouldn't matter much because it can be cleaned up later and because this
function will be called once per repo.
Reviewed By: krallin
Differential Revision: D20311122
fbshipit-source-id: ace2a108790b1423f8525d08bdea9dc3a2e3c37c
Summary:
When checking to make sure the destination directory does not look mounted,
ignore ENOTCONN errors. This can happen if the directory is from a previous
EdenFS instance and did not get unmounted from the kernel properly.
While ideally it would be nicer to run `umount -lf` to clean up this stale
mount point, simply mounting a new checkout on top of this directory is fine.
We don't try to clean up the old mount point here since `umount -lf` would
require root priviliges.
Reviewed By: wez
Differential Revision: D20313117
fbshipit-source-id: c25c85d6b80750662b301f0dff8f293299770a4d
Summary:
Add a command that performs various network tests to check the connection to
the Mercurial server for SSH peers.
These tests are:
* Check we can look up the server name in DNS.
* Check we can make a TCP connection to the SSH server.
* Check we can connect and authenticate over SSH and run `hostname`.
* Check we can make a Mercurial wireproto connection and find the `master` bookmark.
* Check the download and upload speed to the server.
Checking the speed requires a companion speed test program on the server.
Reviewed By: mitrandir77, farnz
Differential Revision: D20305227
fbshipit-source-id: adba02a6a1c40ffc20d7bf9d962a5fcf85e06544
Summary:
Instead of returning `anyhow::Error` wrapping an `ErrorKind` enum
from each Thrift client method, just return an error type specific
to that method. This will make error handling simpler and less
error-prone by removing the need to downcast the returned error.
This diff also removes the `ErrorKind` enums so that we can be sure
that there are no leftover places trying to downcast to them.
(Note: this ignores all push blocking failures!)
Reviewed By: dtolnay
Differential Revision: D20260398
fbshipit-source-id: f0dd96a7b83dd49f6b30948660456539012f82e6
Summary: Pass a valid ObjectFetchContext down into certain untracked requests.
Reviewed By: chadaustin
Differential Revision: D20243575
fbshipit-source-id: e7112c3bab1265803a26130c4d72905c25f2e729
Summary: Make a bytestring with 'b""' to fix Python3
Reviewed By: DurhamG
Differential Revision: D20287313
fbshipit-source-id: 7455d1509684bfb0857a3b060bdcca7e658343fd
Summary: We need to encode/decode utf8 when reading/writing to the hgrc file.
Reviewed By: DurhamG
Differential Revision: D20286068
fbshipit-source-id: b1d6828fb62a83ad22414de6883004411f302142
Summary:
Previously we could have "Started ..." before "Starting ..."
This diff fixes it.
Reviewed By: krallin
Differential Revision: D20277406
fbshipit-source-id: 3c2f3fa1723c2e0852c6b114592ab7ad90be17ff
Summary:
We saw >10 timeouts on Mononoke side from people who fetch a lot of files from corp network (https://fburl.com/scuba/mononoke_test_perf/qd15c5f1). They all tried to download a lot of files, and they 90 mins timeout limit.
Let's try to download these files in rather large chunks (e.g. 200000 files). That should help with resumability and prevent timeout errors, and make retries smaller.
It adds an overhead for establishing a new connection after every 200000 files. That shouldn't be a big problem, and even if it's a big problem we can just increase remotefilelog.prefetchchunksize.
Reviewed By: xavierd
Differential Revision: D20286499
fbshipit-source-id: 316c2e07a0856731629447627ae65c094c6eecc5
Summary: expose the counters for number of pending imports (blobs, trees, prefetches) to allow use in tooling
Reviewed By: chadaustin
Differential Revision: D20269853
fbshipit-source-id: d2b7e2110520290751699c4a891d41ebd5b374cf
Summary:
On bad network link (such as on VPN), the reliability of the connection to
Mercurial might be fairly flaky. Instead of failing fetching files, let's retry
a bit first, in the hope that the connection will be back by then.
Reviewed By: quark-zju
Differential Revision: D20295255
fbshipit-source-id: d3038f5e4718b521ae4c3f2844f869a04ebb25e3
Summary:
The old code does "read, lock, write", which is unsound because after "lock"
the data just read can be outdated and needs a reload.
Reviewed By: xavierd
Differential Revision: D20306137
fbshipit-source-id: a1c29d5078b2d47ee95cf00db8c1fcbe3447cccf
