Summary:
This diff finally introduce the continuous reverse filler. Specifically, this adds a cli (and underlying wiring) to operate the filler logic in the `reversefillerqueue` table.
To achieve this:
- the filler class is turned into a base class with two subclasses for the forward and reverse fillers
- the main file is renamed from `forwardfiller.py` into `filler.py`, to better reflect the independence of direction.
Reviewed By: krallin
Differential Revision: D21628259
fbshipit-source-id: 5676a162a62f0dc6fe80e6300b72d30370fc80b4
Summary: Add devdb support to integration test runner so that one can more easily inspect mysql test data, also makes it easier to run tests with locally modified schema.
Reviewed By: HarveyHunt
Differential Revision: D21645234
fbshipit-source-id: ec75d70ef59f04548c7346a122298567dd09c264
Summary:
1. Add directory existence checking logic before checking "legacy_bind_mounts_dir"
2. Still print the disk consumption returned by the du command when it fails with an error invoking du -skxc
Reviewed By: chadaustin
Differential Revision: D21683421
fbshipit-source-id: 531b6b289e9ffa445ed691611c9cbf22f393e742
Summary:
At first glance people will assume that changesets are returned in the same
order that they were added in the database or that at least commits are
returned in a deterministic fashion. That didn't happen because the both
changeset ids and changeset entries were received without any order.
This diff updates the function to returns results in order they were added
to the database.
Reviewed By: krallin
Differential Revision: D21676663
fbshipit-source-id: 912e6bea0532796b1d8e44e47d832c0420d97bc1
Summary:
This diff is auto-generated to upgrade the Pyre version and suppress errors in eden. The upgrade will affect Pyre local configurations in the following directories:
```
eden
```
Differential Revision: D21687853
fbshipit-source-id: baf0d9bc33f86da63ea289690faca6cf4d566588
Summary:
This structure has similar functionality to the IdMap that is backed by SQL.
It is probably going to be useful for caching in the case of batch operations.
Reviewed By: quark-zju
Differential Revision: D21601820
fbshipit-source-id: 9c3ebc3e9dc92a59ce0908fc241bd2b97da88dca
Summary:
`Dag::build_all_graph` will load the whole graph for a given repository and
construct the segmented changelog from it.
Reviewed By: StanislavGlebik
Differential Revision: D21538029
fbshipit-source-id: b4ba846bb2870ba73257bed6128b8e198a0aab3e
Summary: This function won't go ask remote server and is able to produce a result very quick. This avoids the waiting time in the queue.
Reviewed By: chadaustin
Differential Revision: D21586511
fbshipit-source-id: db3793cc4e4e493d1beb6837c0bd91358e7e845f
Summary:
xavierd and I realized that memcache is not actually enabled in EdenFS as `ContentStoreBuilder` only sets it when there is are remote store. We believe turning on memcache store can help us improve some performance for importing blobs (80% hit rate).
This diff adds a noop remote store that does nothing to trick `ContentStoreBuilder` to include the memcache store. We decided to go this way instead of modifying `ContentStoreBuilder` is because EdenFS is going to have a real remotestore very soon, and changes to `ContentStoreBuilder` may have some impact on how Mercurial uses it.
Reviewed By: xavierd
Differential Revision: D21621234
fbshipit-source-id: 5502e001ab498134b6a927d83be823261e70e9e1
Summary: Reading from hgcache directly has been rolled out for quite a while now. Let's remove the configuration and make it default behavior.
Reviewed By: chadaustin
Differential Revision: D21585482
fbshipit-source-id: 7f30e262642adf58388a0580aa5a63595fa89155
Summary:
gcc and clang disagree here, so explicitly construct a Future from the
unique_ptr.
Reviewed By: genevievehelsel
Differential Revision: D21687654
fbshipit-source-id: 338174f5d4d051f509faace6e48d1c0fb4d898e1
Summary:
There was a bug where if the dynamicconfig chose a value that matched
one of the ported rc files, but it did not match the final value of that config
(from a not-yet-ported rc file), it would chose the dynamicconfig value instead
of the original final value. Let's drop the dynamicconfig value in this case as
well.
Reviewed By: quark-zju
Differential Revision: D21674469
fbshipit-source-id: efcd36e9602e16210999ec8c88e86b1d7ee355e4
Summary: I've updated most messages to suggest a graceful restart, but I overlooked this instance and we got a user question about this. So lets update this message here as well.
Reviewed By: chadaustin
Differential Revision: D21681035
fbshipit-source-id: 32bf5e75b9ad1d98bcd70427101aabeee0be5046
Summary: I didn't update it on my last diff.
Reviewed By: xavierd
Differential Revision: D21687344
fbshipit-source-id: 01c9662b9fd8c4670c2af34a999c69e6b93ed7f7
Summary:
When markbt converted histedit to be compatible with new style mutation
he missed a few spots.
This diffs adds a large test checking the that all the scenarios that were
supported with obsmarkers are supported with mutation now.
Reviewed By: markbt
Differential Revision: D21665484
fbshipit-source-id: a8a289910adb068a382e32f9d106dedc1573d413
Summary:
The histedit goes down to the latest branching point by default which is not
always the right choice for `amend --to`
Reviewed By: markbt
Differential Revision: D21665485
fbshipit-source-id: d981d2216b5384f61c0082684eafa1e84eaeada8
Summary:
This test addtion showcases a bug in `amend --to` when
a side branch is present
Reviewed By: markbt
Differential Revision: D21665486
fbshipit-source-id: 8c21e7a8cccfc6c313d3b7aa09be8309a662b9cc
Summary:
Change from CHashMap to DashMap for the walk state tracking.
DashMap is using slightly less memory in testing, and is slightly quicker in walk rate (number of graph nodes processed per second ).
Reviewed By: HarveyHunt
Differential Revision: D21662210
fbshipit-source-id: ea0601df17e0e596fd59b67d9d01d0dc4e90799b
Summary:
CacheBlob logs hit and miss stastics to Scuba. Let's add the same for
ODS.
Reviewed By: krallin
Differential Revision: D21640922
fbshipit-source-id: 8f7d17f048bf53bdc6cd8bda0384a51cae7b6a30
Summary:
CountedBlobstore is a Blobstore that wraps another blobstore in order
to report metrics about it, such as the number of gets or errors. It's commonly
used to wrap a CacheBlob blobstore, which itself is a caching wrapper over an
inner blobstore.
CountedBlobstore exposes metrics that are supposed to track the number of hits
or misses when fetching blobs. However, these metrics don't make sense as the
CountedBlobstore has no view into cache activity. These metrics actually report
the number of requests and the number of missing blobs rather than hits and
misses.
Remove these misleading counters.
Reviewed By: krallin
Differential Revision: D21640923
fbshipit-source-id: 07b9fc9864c70991415c2b84f35d631b702c17d1
Summary:
now fastlog will be able to show a history of file like that:
```
Added
|
Deleted
|
Added
```
We do it by using deleted file manifest to find where a file was deleted. We do
it in case we find a unode which doesn't have a parent.
Differential Revision: D21664248
fbshipit-source-id: 8e80fccc87361e66e4e00ae2da3671ed8327f1f8
Summary:
... or rather make this special case a bit smaller.
In fastlog we have a `prefetch` which was already returned to the caller,
however its parents hasn't been processed yet. Previously we had two separate
places where these parents were processed, now we have only one.
This will make it easier to add support for history across deletions in the
next diff.
Differential Revision: D21664246
fbshipit-source-id: a5a109fafaeec28a0e7a36ffa44545aebc032b00
Summary: I had hardcoded the path separator, which is wrong on Windows.
Reviewed By: singhsrb
Differential Revision: D21679423
fbshipit-source-id: 309c84b0698355654baee1dae7f201310e695c49
Summary:
Just top-level mononoke and cache_warmup as a more or less random
selection.
Reviewed By: StanislavGlebik
Differential Revision: D21648852
fbshipit-source-id: 0e5137da9a7742377112e085e9d29182abdfbf41
Summary: Fix some RVO warnings and a compilation error when compiling with gcc.
Reviewed By: xavierd
Differential Revision: D21481738
fbshipit-source-id: 0621f5886df40c24ef1a6a68ccd957e38f2f4122
Summary:
If either the config, or the environment variable is empty, the URL parsing
would raise an error, while the intent is to not have proxy. Let's handle it
properly.
Reviewed By: DurhamG
Differential Revision: D21669922
fbshipit-source-id: 84c8d09242ac8d5571f7592d874f4954692110f5
Summary:
Looks like echo prints a \r\n which messes up the output. Let's use
printf to force it to use \n
Reviewed By: xavierd
Differential Revision: D21670125
fbshipit-source-id: f991c9c9fb55720bb68a3f71e8ddd3b4f16b5375
Summary:
At clone time we apply dynamic configs in memory, instead of loading
them from disk. The validation logic operated on the config value's location
field, which isn't set for data that comes from in memory. So we need to update
the validation logic to also consider the value source if location is not set.
Reviewed By: quark-zju
Differential Revision: D21664205
fbshipit-source-id: 8460c58c6d654780048de51ada8178c70ff0a9e6
Summary:
We kick off a background process every 15 minutes or so to compute the
new dynamicconfig. If the config hasn't changed, we should just touch the file
instead of rewriting the whole thing.
Reviewed By: quark-zju
Differential Revision: D21653098
fbshipit-source-id: 9d53d2a636cff082cd048994255cc809ce1b0221
Summary:
If we release a new version of Mercurial, we want to ensure that it's
builtin configs are used immediately. To do so, let's write a version number
into the generated config file, and if the version number doesn't match, we
force a synchronous regeneration of the config file.
For now, if regeneration fails, we just log it. In the future we'll probably
throw an exception and block the user since we want to ensure people are running
with modern configuration.
Reviewed By: quark-zju
Differential Revision: D21651317
fbshipit-source-id: 3edbaf6777f4ca2363d8617fad03c21204b468a2
Summary:
Now that regular users can create symlinks, a handful of tests were failing
due to the symlink call no longer raising an exception. For now, let's always
consider that symlinks aren't supported on Windows.
Reviewed By: fanzeyi
Differential Revision: D21664213
fbshipit-source-id: c55a99d1cb92e68b9861701b5517b1d5db2d40c6
Summary:
Previously we created `visited` and immediately drop it.
Let's instead create it one level up. I don't think it'd matter in practice,
but it should be cleaner
Reviewed By: HarveyHunt
Differential Revision: D21662456
fbshipit-source-id: 36823309ea09448c5be10c2655ea0f317649f290
Summary:
Follow up from D21596758 - current logic of pushrebasing a merge is very
complicated. To prevent repo corruptions let's do a very simple validation -
generate hg changeset from a rebased bonsai changeset.
Given that generating hg changeset is an expensive operation let's do it only
after the first rebase attempt - otherwise we might risk constantly losing the
pushrebase race.
Differential Revision: D21659187
fbshipit-source-id: f43a855cf0fbdbd11a40d3ec38283af344cde5e6
Summary: This hint has served its purpose. Remove it.
Reviewed By: farnz
Differential Revision: D21658927
fbshipit-source-id: 0395fa2cb898732b2c64154104c703a428f62864
Summary: This is to bring it into sync with the `forwardfillerqueue` types.
Reviewed By: markbt
Differential Revision: D21660012
fbshipit-source-id: 5148023478c175cd49707d88251701a08fcbe0ce
Summary:
Mononoke server is the only binary using plain `clap::App`, whilst all
other services use `MononokeApp`. The `MononokeApp` variant configures some
default arguments and provides consistent argument naming between binaries.
Update the server to use `MononokeApp` and add some logic to allow passing
either `--config_path` or `--mononoke-config-path` until we remove all uses of
`--config-path`.
Reviewed By: StanislavGlebik
Differential Revision: D21642461
fbshipit-source-id: f42e0b6625f3979ced0920db269bdb4528f99e49
Summary:
When starting up with many repos, it can take a while before the server is ready
to accept connections, and there is no log to indicate when that is the case.
Add one.
Reviewed By: farnz
Differential Revision: D21641189
fbshipit-source-id: 057ca06c1b1725799a8445841b6eea89bba783b0
Summary:
Rather than logging `Refreshing tunables` all the time, log only when they
change, and include the values that they are changing from and to.
Reviewed By: farnz
Differential Revision: D21641191
fbshipit-source-id: b0b5edd687e2e41074ce1c8be13fd6918dff924d
Summary:
There are a few paths through `resolve` that return early with a failure, and thus never record what happened.
Make a record the moment we enter `resolve` - then, we can use `count` type ODS charts to determine the failure rate deterministically, and alarm if the failure rate is too high
Reviewed By: ahornby
Differential Revision: D21647575
fbshipit-source-id: 667787ec000a8cd8e715563df10dbb84832fefa1
Summary:
During clone the hgrc.dynamic file doesn't exist and doesn't even have
a place for us to generate it to. Let's instead generate and apply the config in
memory.
In the future, if generate fetches data from the network, this will mean clone
would depend on the network, since if generate fails the clone would fail. In
some situations this is desirable, since users shouldn't be cloning without our
approved configs, but if it causes problems we could probably tweak generate to
support an offline mode.
Reviewed By: quark-zju
Differential Revision: D21643086
fbshipit-source-id: d9a758207738d5983213d95725061517e0aa17db
Summary:
The Rust version command does not support `-T` and does not read
`__version__.py`. Remove tests about them. As we're here, also remove tests
about 3rd-party extension version checking as we bundle all extensions.
Reviewed By: xavierd
Differential Revision: D21648591
fbshipit-source-id: 97c9e2ff3d224ff12b34801a1af07532d9e9bff5
Summary:
Implements a "debugdetectissues" command, which runs a series of checks on the repository meant to detect potential issues, logging the data to Scuba so that we can determine how common certain issues are, and under what conditions they appear.
Future extensions of this change may involve merging the functionality into hg doctor, and setting the command to run automatically in the background on some interval.
Reviewed By: DurhamG
Differential Revision: D21558170
fbshipit-source-id: a878ae1804d5f11c83a574e0dc3c802b564d2ced
Summary:
This script deals with obtaining the right privilege to start the EdenFS
service. This effectively enables `edenfsctl start`.
Reviewed By: wez
Differential Revision: D21585739
fbshipit-source-id: 7b835434d865fa4c4c8473e13665ae669fd86108
Summary: Now that Rust libraries can be linked into fbcode binaries with mode/mac, enable Rust datapack on all platforms.
Reviewed By: xavierd
Differential Revision: D21241075
fbshipit-source-id: ae2abee25a5ad7b9db9ac5e6b8687c5f79376926
Summary:
There is no need to upload content referred by a public commit.
This affects cases like `debugstrip` (ex. for testing `pull` performance with a
lagged commit graph). Without this change, the uploadrevs code path scans
stripped public commits without efficient tree prefetching, which results in
1-by-1 tree fetches and is unusably slow.
Reviewed By: xavierd
Differential Revision: D21630096
fbshipit-source-id: 385edf76cb4eb913b2d64422910cdb46b603e6c0
Summary:
With `HGPLAINEXCEPT=color`, colors should still be enabled if the terminal is
capable of supporting them and output is to the terminal.
Currently this doesn't work if `--color=auto` (the default), as
`color._modesetup` uses `ui.formatted` to check if the output is a terminal,
and thus has colors available, but this is `False` for all `HGPLAIN` modes.
Instead, add a new method to `ui` that checks whether `fout` is a terminal, and
use that for color autodetection.
This function also allows us to add `HGPLAINEXCEPT=pager` as we can use
that in the same way.
Reviewed By: farnz
Differential Revision: D21617170
fbshipit-source-id: 7ee4eaa8963f3d6eb7ed8044a678a4804b9a98f0
Summary:
Setting the fake pager in the pager test doesn't work on Windows.
For some reason, the `pager.pager=C:/path/to/python.exe C:/path/to/test/fakepager.py` triggers
some kind of PATH translation code that results in `setconfig.py` receiving
`pager.pager=C;C:\\path\\to\\python.exe C;C:\\path\\to\\test\\fakepager.py` as its
argument. This is clearly invalid, so the test fails with messages like
`'C' is not recognized as an internal or external command`.
Workaround this for now by setting the pager by appending directly to HGRCPATH.
Reviewed By: ikostia
Differential Revision: D21638507
fbshipit-source-id: e8a6b0c281030b23302116a79ef3ba754d37f601
Summary:
D21573455 accidentally removed the scuba sampling for scrub_get
operations. Add this back in.
Reviewed By: StanislavGlebik
Differential Revision: D21638972
fbshipit-source-id: eee66dbce161de69246f4da0a15dc2cf00e1ba01
Summary:
See bottom diff in the stack for the motivation. Though you can probably guess
the motivation :)
Reviewed By: farnz
Differential Revision: D21623154
fbshipit-source-id: a0940d766a67080ddcb346c2e3313eb08699edad
Summary: First diff in the stack that removes getfiles since it's no longer needed.
Reviewed By: farnz
Differential Revision: D21623156
fbshipit-source-id: 44f310ec4e4f34845cc5bf1738f1a8ece14e6694
Summary:
Before this change, pull_shallow only calls setuprepo if the remotefilelog
requirement is not in repo. With D21011401, the remotefilelog requirement
will be added by clone.py and pull_shallow can skip calling setuprepo, causing
the pull code paths to write file logs (and fail).
Change the pull_shallow to always call `setuprepo` to solve the issue.
The final fix is probably moving more remotefilelog related clone logic to core.
Right now I just did the minimal change to fix things.
Reviewed By: sfilipco
Differential Revision: D21632429
fbshipit-source-id: 17775ac0df18cda10247419b40f9c27436b22606
Summary:
We didn't have a high level overview of the types and traits and their
purposes, add that.
Reviewed By: DurhamG
Differential Revision: D21599759
fbshipit-source-id: 18e8d23ed00134f9d662778eddaee4a9451f7c2c
Summary: EdenFS doesnt daemonize correctly due to the privhelper not closing fd 0 (see http://www.faqs.org/faqs/unix-faq/programmer/faq/). This redirects stdin to /dev/null/ in order to do so.
Reviewed By: xavierd
Differential Revision: D21602545
fbshipit-source-id: 0aeb589efbf214ef22c0db039fbb6a436a71e360
Summary:
This updates a placeholder in the authentication process for requests
in the rust thrift services to allow authentication with CATs. This change
allows for passing in a configurable server identity to be used as the
verifier identity here.
Additionally this sets the verifer identity to be "scm_service_identity"
for the mononoke scs server (the service identity for this service).
This is needed to use the mononoke scs server from corp.
Reviewed By: markbt
Differential Revision: D21560204
fbshipit-source-id: 02e41718307fddab6f0ac2101306bb7f4a59cf7e
Summary: Otherwise the version information is only available in Python.
Reviewed By: DurhamG
Differential Revision: D19803762
fbshipit-source-id: 044c5da86efc8c657d0c422a2b1947086444895e
Summary:
Currently, we have a gap between updating and subsequently reading the
value of `write_order`. If another blobstore's put operation completes before
we have called `record_put_stats` then we may have an incorrect `write_order`
value.
Whilst this is only a minor issue, let's fix it anyway. :-)
Reviewed By: krallin
Differential Revision: D21619669
fbshipit-source-id: 1b8bacbcb4c195e6765ebdfaa68425f286f88c06
Summary:
After other optimizations the CHashMap version of visit_count was showing as a hot spot.
Given the number of possible NodeTypes is small we can store the visit_count in array indexed by ordinal instead.
Reviewed By: farnz
Differential Revision: D21618518
fbshipit-source-id: 84978778034df11a9a48452adf9269db2dc17145
Summary:
Currently we record them only during pushrebase. Let's record during push as
well.
To simplify things a little bit let's allow only a very simple push case:
1) Single bookmark.
2) All pushed commits should be reachable by this bookmark.
Reviewed By: krallin
Differential Revision: D21451337
fbshipit-source-id: bf2f1e6025ac116fb8096824b7c4c6440d073874
Summary:
This is the fix that might be hard to explain.
First, why we need `generate_additional_bonsai_file_changes()` function at all.
TL;DR is to fix pushrebasing of a diamond merge commit. Merge bonsai changeset
must have a file change entry if p1 and p2 has the same file with different
content, and when merge commit is pushrebased we need to preserve this property.
more details are in D18065013 and in the comments.
Secondly, what was wrong with generate_additional_bonsai_file_changes. While it
was technically correct (i.e. it was producing valid bonsai changeset) it had
an unexpected side-effect for non-diamond merges (e.g. repo imports). It was
adding all changed files between root and onto commit (i.e. all changed files
from commits over which it was rebased). This in turn leads to unnecessary
push failures with "conflicts" errors.
Thirdly, why it was wrong. generate_additional_bonsai_file_changes
a) finds all files changed between onto and root (see comments for more details about onto
and root),
b)then finds those that needs to be added in the pushrebase merge bonsai changeset.
The problem was in step b). It needed to check which files exists in merge
parent but instead it checked which files exists in merge commit itself.
This is because merge commit itself cannot change any of the files that were changed between root
and onto - in that case we get a pushrebase conflict. So all the files come
either from root commit or from another parents. And the entry can become
"stale" only if it existed in one of the parents and later was changed between
root and onto.
Reviewed By: krallin
Differential Revision: D21596758
fbshipit-source-id: f2bea36fcc29fc736caaa7494696a1f6dc848b9e
Summary: Add MPathHash memoization to WrappedPath as the paths are hashed multiple times
Reviewed By: farnz
Differential Revision: D21613720
fbshipit-source-id: b850cf5ea1668b5ff75b07a0489b54f86078677a
Summary:
Currently, there isn't a way to override the value of a tunable during
a unit test. This is unfortunate and may push people to creating integration
tests when unit tests would be best.
Add `with_tunables`, which allows callers to override the tunables values that
a closure accesses. Further, add an example unit test that shows the behaviour.
Thanks to krallin for suggesting this and helping with the implementation
details.
Reviewed By: krallin
Differential Revision: D21595220
fbshipit-source-id: c31da88fb4fcfd38f761960ffe8025132025e6c9
Summary:
Let's add an option to log how many files and trees were fetched in a
particular repo that start with a prefix.
Reviewed By: farnz
Differential Revision: D21617347
fbshipit-source-id: a57f74eadf32781e6c024e18da252c98af21996d
Summary:
Just as for ints and bools, let's add support for strings.
A few notes:
1) we don't have AtomicString, so I decided to use ArcSwap<String>.
However to make code generation simpler (see tunables-derive) I've added a type
alias for that
2) Return type is Arc<String> to avoid unnecessary copies.Another option that we might have here is to put the whole Tunables structure inside ArcSwap, and changing `tunables()` to return Arc<MononookeTunables>
instead of &'static MononokeTunables. However that's a bigger change to make.
Reviewed By: markbt
Differential Revision: D21592390
fbshipit-source-id: 6d3cf340b13f7aef9adb2b1b99ed2bf260033285
Summary:
An earlier change (D21394302) made discovery sampling use the most
recent commits. This ended up not being correct since if the requested sample
did not find any hits then the unknown set would not be reduced and the next
requested set would not change, resulting in an infinite loop.
The goal of the original diff was to ensure master was in the initial sample set
so we didn't have to inspect a bunch of really old commits. Let's go back to
random sampling, but manually insert master into the initial set.
In the long-term we should throw away all this code and use something similar to
discovery.fastdiscovery instead which just relies on the public heads.
Reviewed By: quark-zju
Differential Revision: D21577331
fbshipit-source-id: d738de255e292fc569eb27f250dcd3eaafaaae43
Summary:
This checks in some design documents based on the "EdenFS Internals"
presentation that I've given internally a few times.
Reviewed By: wez
Differential Revision: D21519530
fbshipit-source-id: 3f79d38e8ccf994b2ef303d491809a91fa5b6d95
Summary:
Next step in unifying the mount path, let's make the initialize the same in
Windows and unices. The only difference is now limited to the .eden directory
which we will be able to implement once regular users can create symlinks.
For the takeover code, the #ifdef is pushed down to the actual code that does
it, this allows the rest of the code to not have to bother about Windows vs
other platforms.
Reviewed By: wez
Differential Revision: D21517478
fbshipit-source-id: d40ca2694d23031ff98e319071e610efa306008f
Summary:
The blobstore multiplexer contains logic to log blobstore operations to
scuba, as well as updating `PerfCounters`. There are some cases where we don't use the
multiplexed blobstore, which means that we're missing this important logging.
Factor out the logging to a separate crate and implement `LogBlob`, which wraps
another blobstore and performs logging to both scuba and PerfCounters.
Reviewed By: StanislavGlebik
Differential Revision: D21573455
fbshipit-source-id: 490ffd347f1ad19effc93b93f880836467b87651
Summary:
The configerator thrift file was updated to remove a path from hook
config. However, this change wasn't synced to fbsource.
Sync the change and fix up the tests that are broken by this change.
Reviewed By: krallin
Differential Revision: D21594221
fbshipit-source-id: 7b64180914f6c6802e4d70fcb1a5d6ec36eb2eac
Summary:
When we don't provide a repo name, scm_telem_log will run `hg config` to try
and get the path, and fall back to using the basename of the repo directory.
However, this is a bit undesirable, because if we ran this code we already
checked the path, so the repo URL isn't going to magically materialize once `hg
config` asks for it, which means we make a completely redundant call to hg from
scm_telem_log.
By just doing this in logginghelper, we avoid this extra roundtrip.
Reviewed By: StanislavGlebik
Differential Revision: D21572027
fbshipit-source-id: 58e5ab2e3e525edef1ecde039cd968eab8d89172
Summary:
My jq has 2 spaces after "Usage:", so this fails. Let's make the matching a bit
more lenient.
Reviewed By: StanislavGlebik
Differential Revision: D21572028
fbshipit-source-id: 1948af8bbee6486df78299c3cc08209ce3ef3e12
Summary: This might provide some useful insights for debugging hanging processes.
Reviewed By: kulshrax
Differential Revision: D21580307
fbshipit-source-id: 424992cdd9851bfd782a3333ed22ac1193c33fe3
Summary:
Make `prev` and `next` match the behavior of `update` and allow movement with pending changes. A new `--check` flag has been added to enforce a clean working copy (similar to `update`).
Note that these commands literally just call `update` under the hood, so this was just a matter of removing the existing dirtiness check. (The check was originally put there because `update` originally required a clean working copy by default.)
Reviewed By: quark-zju
Differential Revision: D21581805
fbshipit-source-id: 33d0ac4d36a795713054af7e8776d077fd353048
Summary: Previously, we would only allow BFS prefetching when HTTP was enabled (since prior to the `designatednodes` server capability, SSH had no way of supporting BFS). That has now changed, so we should allow BFS even if HTTP is disabled.
Reviewed By: quark-zju
Differential Revision: D21581215
fbshipit-source-id: b9d5945730e2c718681004f7d5e5f319449fcf6a
Summary:
Now we have a virtual DAG that can be queried. Use that to figure out what
heads to hide and add instead of manually traversal through the graph.
Also did some tweaks to fix Python 3 compatibility and make the parent graph
style a bit more correct.
Reviewed By: markbt
Differential Revision: D21554672
fbshipit-source-id: 749d7938a8612e21c5975d9b80a275a059de022d
Summary:
The `getsmartlog` APIs return a generator suitable for passing to the graphlog
rendering layer, but not suitable for doing other things such as querying the
DAG. Split that API into two APIs:
- getsmartlog -> SmartlogInfo
- makedagwalker(SmartlogInfo, Repo) -> generator suitable for graph rendering
Reviewed By: markbt
Differential Revision: D21554673
fbshipit-source-id: c709b7a41572eb3e9c9f91b74fc2bc8226bc481f
Summary:
This simplifies a lot of the code, and makes it possible to do DAG queries like
ancestors, descendants on the commit cloud graph.
Reviewed By: markbt
Differential Revision: D21554674
fbshipit-source-id: ee08cddfc162a7546d63d4bf385f2948fc799fd3
Summary:
The only reason that we keep the legacy renderer is that `hg-sl-up` parses
`hg sl` output and breaks with the new Rust renderer. `cloud sl` does not
have such issues so we can switch to the new renderer unconditionally.
Together with the previous change, this allows the upcoming change to render
the graph without using revision numbers.
Reviewed By: markbt
Differential Revision: D21554670
fbshipit-source-id: eee5fbd641096b38e206298f9e716df358a7ab7e
Summary:
Change pyrenderdag to accept non-revision-number graph vertexes so it can
render a graph even if the graph does not use revision numbers.
The next diff wants this behavior so it can just emit commit hashes to
the renderer without caring about revision numbers. The type is made
so it can still support revision numbers, since the legacy graphlog
interface would still use revision numbers.
Reviewed By: markbt
Differential Revision: D21554671
fbshipit-source-id: 20572683b831f7cecb03957c83f278ff3903eff0
Summary:
The previous code was wrong - it converts the PyObject to iterator every time
(ex. if the PyObject is a set, then it calls `set.__iter__` every time, and
will only get the first element of the set).
For example, it will enter an infinite loop for evaluating this:
bindings.dag.nameset({'1', '2'})
Fix it by calling `__iter__`, to get the iterator object and use that instead
of the original PyObject.
Reviewed By: markbt
Differential Revision: D21554676
fbshipit-source-id: 0f2adae8f123530cee2d473da37ca1a93a941fde
Summary:
This function reorders commits so the graph looks better.
It will be used to optimize graph rendering for cloud smartlog (and perhaps
smartlog in the future).
Reviewed By: markbt
Differential Revision: D21554675
fbshipit-source-id: d3f0f27c7935c49581cfa6e87d7c32eb5a075f75
Summary: This makes it easier to do `a & b`, `a | b`, `a - b`.
Reviewed By: markbt
Differential Revision: D21554677
fbshipit-source-id: e1e2571a3dc83f80a1ec7a056f2c8f71ab292d9e
Summary:
This adds support for periodically logging that a command is in progress in
Mononoke. The underlying motivation is to make sure that if something is taking
a long time, we can still show some feedback to the user (and log to Scuba). We
might want to log this every 30 seconds.
That said, this is more of an RFC at this stage. I'm thinking it might make
sense to log to Scuba more often and to users less often. It might make sense
to also restrict this to specific commands, such as unbundle:
https://fburl.com/scuba/mononoke_test_perf/atik5959
Reviewed By: StanislavGlebik
Differential Revision: D21549862
fbshipit-source-id: 1d02c5c926abc7e491ac5b8ae0244b5f4620c93e
Summary: Same as the previous diff, but for commands that return a stream.
Reviewed By: StanislavGlebik
Differential Revision: D21549864
fbshipit-source-id: ba8c14db34a651cd4ddbc1c8b9ad382c08cc775d
Summary:
This doesn't do anything on its own, but it's refactoring I need for later in
this stack. It wraps all our commands in a command_future call that gives us an
opportunity to wrap the future being returned. We used to use `start_command`
to get the context, so this just replaces that.
Reviewed By: StanislavGlebik
Differential Revision: D21549863
fbshipit-source-id: 0e613bb1db876d27d662fd6c993d7b7d954b5f2b
Summary:
This allows us to find the closest globalrev (i.e. the highest that is lower
than or equal to the one being requested). Conceptually, this is similar to
prefix lookups, which is why we're including it in the prefix lookup function
method!
Reviewed By: markbt
Differential Revision: D21553081
fbshipit-source-id: 4b08f7e2478210d406b1a043e95f8768a3dcc45a
Summary:
The output streams from the walks are of the form (Key, Payload, Stats).
Where the Key is Node and the Payload is NodeData this is ok, but with the key and payload both tuples it gets hard to read, so this introduces named tuple-like structs for clarity.
Reviewed By: StanislavGlebik
Differential Revision: D21504916
fbshipit-source-id: a856d34af4117d3183ef0741b311c1c34cf9dacc
Summary:
Add a --sample-path-regex option for use in the corpus dumper so we can dump out just a subset of directories from a repo.
This is most useful on large repos.
Reviewed By: farnz
Differential Revision: D21325548
fbshipit-source-id: bfda87aa76fbd325e4e01c2df90b5dcfc906a8f6
Summary:
Track path mtime, as being able to order by mtime is important to be able to use the on disk corpus to evaluate delta compression approaches
The dumped blobs mtime is set based on the last traversed bonsai or hg commit's timestamp. For Bonsai it prefers committer_time if present and if not falls back to author_time.
Reviewed By: farnz
Differential Revision: D21312223
fbshipit-source-id: fa14615603f78675ca54a0f4946cc8480b8eade5
Summary:
Update the corpus walker to dump the sampled bytes as early as possible to the Inflight area of the output dir, then move them to final location once path is known.
When walking large files and manifests this uses a lot less memory that holding the bytes in a map!
Layout is changed is to make comparison by file type easier. we get a top level dir per extension, e.g. all .json files are under FileContent/byext/json
This also reduces the number of bytes taken from the sampling fingerprint used to make directories, 8 was overkill. 3 is enough to limit directory size.
Reviewed By: farnz
Differential Revision: D21168633
fbshipit-source-id: e0e108736611d552302e085d91707cca48436a01
Summary:
Add corpus dumper for space analysis
This reuses the path based tracking from compression-benefit and the size sampling from scrub.
The core new functionality is the dump to disk from inside corpus stream.
Reviewed By: StanislavGlebik
Differential Revision: D20815125
fbshipit-source-id: 01fdc9dd69050baa8488177782cbed9e445aa3f7
Summary:
Make it so that remote bookmarks like `foo/name` or `bar/name` would pull from
different sources `paths.foo` or `paths.bar`.
Reviewed By: DurhamG
Differential Revision: D21526666
fbshipit-source-id: 6791ab047840c6c49df0c96ff1f56ae7bd1aeeba
Summary:
We now have auto pull logic that covers most unknown rev use-cases. The hint
message is no longer necessary. It's also unclear how to use `hg pull`
correctly. For example, should it be `-r`, `-B remote/foo` or `-B foo`?
Reviewed By: DurhamG
Differential Revision: D21526667
fbshipit-source-id: 40583bfb094e52939130250dd71b96db4d725ad5
Summary: The original purpose of this test was to verify that `hg restack` would work correctly with the `inihibt` extension disabled. `inhibit` has not been relevant at FB for years, so this test has no value.
Reviewed By: quark-zju
Differential Revision: D21555411
fbshipit-source-id: 475ed37439ed71aee08ad1b23ebe1770c3324890
Summary:
I want to reuse the functionality provided by `fetch_all_public_changesets`
in building Segmented Changelog. To share the code I am adding a new crate
intended to store utilities in dealing with bulk fetches.
Reviewed By: krallin
Differential Revision: D21471477
fbshipit-source-id: 609907c95b438504d3a0dee64ab5a8b8b3ab3f24
Summary:
This is helpful, when we have raw unbundle bytes and a server path and just
want to send these bytes server's way.
Very similar to `sendunbundlereplay`, but does not do anything additional,
and reads from stdin.
Reviewed By: markbt
Differential Revision: D21527243
fbshipit-source-id: 97726cb40a32c7e44f47e0f56d8c8eabc4faf209
Summary:
As a developpers is working on large blobs and iterating on them, the local LFS
store will be growing significantly over time, and that growth is unfortunately
unbounded and will never be cleaned up. Thankfully, one the guarantee that the
server is making is that an uploaded LFS blob will never be removed[0]. By using
this property, we can simply move blobs from the local store to the shared
store after uploading the blob is complete.
[0]: As long as it is not censored.
Reviewed By: DurhamG
Differential Revision: D21134191
fbshipit-source-id: ca43ddeb2322a953aca023b49589baa0237bbbc5
Summary:
In order to build a StringPiece from a C-string, the length of the that
string needs to be known, and a constexp strlen is performed on it. That
strlen is however a recursive one, causing the stack to blow up on big file.
Interestingly enough, this also means that binary files potentially had a
wrong sha1 being computed, potentially causing `hg status` to report some
files as being modified, while they aren't. By switching to using a vector
instead of a string, the intent should of this should be more obvious.
Reviewed By: simpkins
Differential Revision: D21551331
fbshipit-source-id: 2dc1f08d96f49d310593e0e934a03215be2b5cbb
Summary:
If a push or pull operation doesn't involve any changesets for which mutation
information might be relevant, don't spend any time querying the database, and
instead exit early.
Reviewed By: krallin
Differential Revision: D21549937
fbshipit-source-id: a6f992e621456b826acd1bddde3591e751d23b31
Summary:
MySQL returns an error for a query of the form `WHERE value IN ()`. Avoid
these by checking that collections are not empty before making the query.
Reviewed By: krallin
Differential Revision: D21549690
fbshipit-source-id: 1507d36e81f7a743d2a1efb046e52a5479633ab9
Summary:
The `test-infinitepush-mutation.t` test covers the new mutation database, so
add it to the mysql tests.
Reviewed By: krallin
Differential Revision: D21548966
fbshipit-source-id: 0dc1f90129fa61fb6db1c1b5a747efa3d20041f5
Summary:
The `support_bundle2_listkeys` flag controls at runtime whether we support
`listkeys` in bundle2. Since this was added before tunables were available,
it uses a value in the mutable counters SQL store.
We could migrate this to tunables, but in practice we have never disabled it,
so let's just make it the default.
Reviewed By: krallin
Differential Revision: D21546246
fbshipit-source-id: 066a375693757ea841ecf0fddb0cc91dc144fd6f
Summary:
When the client pulls draft commits, include mutation information in the bundle
response.
Reviewed By: farnz
Differential Revision: D20871339
fbshipit-source-id: a89a50426fbd8f9ec08bbe43f16fd0e4e3424e0b
Summary:
Advertise support for `b2x:infinitepushmutation`. When the client sends us
mutation information, store it in the mutation store.
Reviewed By: mitrandir77
Differential Revision: D20871340
fbshipit-source-id: ab0b3a20f43a7d97b3c51dcc10035bf7115579af
Summary: Add the mutation store to blobrepo.
Reviewed By: krallin
Differential Revision: D20871336
fbshipit-source-id: 777cba6c2bdcfb16b711dbad61fc6d0d2f337117
Summary: What it says in the title
Reviewed By: StanislavGlebik
Differential Revision: D21549635
fbshipit-source-id: 75939ebbfb317a9beaa9acd1fc1a7c6f41b0f88f
Summary: The compiler is warning about it.
Reviewed By: singhsrb
Differential Revision: D21550266
fbshipit-source-id: 4e66b0dda0e443ed63aeccd888d38a8fcb5e4066
Summary: Expose the API that returns a real graph.
Reviewed By: DurhamG
Differential Revision: D21486520
fbshipit-source-id: 4ebdb4011df8971c54930173c4e77503cd2dac47
Summary:
Part of the mutation graph (excluding split and fold) can fit in the DAG
abstraction. Add a method to do that. This allows cross-dag calculations
like:
changelogdag = ... # suppose available by segmented changelog
# mutdag and changelogdag are independent (might have different nodes),
# with full DAG operations on either of them.
mutdag = mutation.getdag(...)
mutdag.heads(mutdag.descendants([node])) & changelogdag.descendants([node2]) # now possible
Comparing to the current situation, this has some advantages:
- No need to couple the "visibility", "filtered node" logic to the mutation
layer. The unknown nodes can be filtered out naturally by a set "&"
operation.
- DAG operations like heads, roots can be performed on mutdag when it's
previously impossible. We also get operations like visualization for free.
There are some limitations, though:
- The DAG cannot represent non 1:1 modifications (fold, split) losslessly.
Those relationships are simply ignored for now.
- The MemNameDag is not lazy. Reading a long chain of amends might be slow.
For most normal use-cases it is probably okay. If it becomes an issue we
can seek for other solutions, for example, store part of mutationstore
directly in a DAG format on disk, or have fast paths to bypass long
predecessor chain calculation.
Reviewed By: DurhamG
Differential Revision: D21486521
fbshipit-source-id: 03624c8e9803eb1852b3034b8f245555ec582e85
Summary:
subcommand_single calls `derived_data_utils.regenerate(vec![cs_id])` with the
intention that derived data for this commit will be regenerated. However
previously it didn't work because DerivedDataUtils::derive() was ignoring
regenerate parameter. This diff fixes it.
Reviewed By: krallin
Differential Revision: D21527344
fbshipit-source-id: 56d93135071a7f3789262b7a9d9ad84a0896c895
Summary:
This allows us to replace the pyre2 C++ bindings so the fast regex engine can
work with Python 3, and simplify our build steps.
Reviewed By: DurhamG
Differential Revision: D20973179
fbshipit-source-id: e123ac18954991f2c701526108f5c2ecd2b31a3b
Summary: Add a `/history` endpoint that serves EdenAPI history data. Like the other endpoints, this one currently buffers the response in memory, and will be modified to return a streaming response in a later diff.
Reviewed By: krallin
Differential Revision: D21489463
fbshipit-source-id: 259d2d1b7d700251fe902f1ac741545e5895404a
Summary: Add the ability to parse EdenAPI history responses to `data_util`.
Reviewed By: quark-zju
Differential Revision: D21489228
fbshipit-source-id: 42dda64273673431a6f3e4d7bd430689c76c387f
Summary: Factor out logic that will be common to many handlers into new functions in the `util` module.
Reviewed By: krallin
Differential Revision: D21489469
fbshipit-source-id: 9aff4e5182748ab0a0bedd6038852692b8e721a7
Summary: Break up the EdenAPI server integration tests to prevent the test from getting too long.
Reviewed By: krallin
Differential Revision: D21464056
fbshipit-source-id: 076aaf8717547fe9188f40c078d577961c02325d
Summary: Add an endpoint that serves trees. Uses the same underlying logic as the files endpoint, and returns the requested nodes in a CBOR DataResponse.
Reviewed By: krallin
Differential Revision: D21412987
fbshipit-source-id: a9bcc169644a5889c3118a3207130228a5246b2f
Summary: Change `make_req` to take a JSON array as input when constructing `DataRequest`s instead of a JSON object. This is more correct because DataRequests can include multiple `Key`s with the same path; this cannot be represented as an object since an object is effectively a hash map wherein we would have duplicate keys.
Reviewed By: quark-zju
Differential Revision: D21412989
fbshipit-source-id: 07a092a15372d86f3198bea2aa07b973b1a8449d
Summary: EdenAPI data responses can contain data for either files or trees. As such, the implementation of both the files and trees endpoints is almost identical. To allow the logic to be shared between both, this diff makes the handler code generic.
Reviewed By: krallin
Differential Revision: D21412986
fbshipit-source-id: 89501915b0401214b373ed1db2eb09e59de2e5b7
Summary: In order to allow writing code that is generic over files and trees, move the functionality common between the two to a separate trait. This will allow for a significant amount of code sharing in the EdenAPI server. (This diff does not introduce any new functionality; it's mostly just moving existing code into the new traits.)
Reviewed By: krallin
Differential Revision: D21412988
fbshipit-source-id: 31b55904f62ccb965b0f9425de875fc069e10b5a
Summary:
Add an endpoint that serves Mercurial file data.
The data for all files involved is fetched concurrently from Mononoke's backend but in this initial version the results are first buffered in memory before the response is returned; I plan to change this to stream the results in a later diff.
For now this version demonstrates the basic functionality as well as things like ACL enforcement (a valid client identity header with appropriate access permissions must be present for requests to succeed).
Reviewed By: krallin
Differential Revision: D21330777
fbshipit-source-id: c02a70dff1f646d02d75b9fc50c19e79ad2823e6
Summary: Nothing should be running with this version, so this check is not needed anymore!
Reviewed By: chadaustin, wez
Differential Revision: D21513269
fbshipit-source-id: 698c9d84e87461cd0484ef997f44dc9169e87a25
Summary:
While not all stats are supported on Windows, most of them can, let's enable
them. This is a first step in unifying the mount path for Windows and Unices.
Reviewed By: chadaustin
Differential Revision: D21514732
fbshipit-source-id: 00fd89b7499f8ae18c089f98bc18cd43f68538fa
Summary:
Write commit cloud sync and clienttelemetry blackbox logs to tracing data.
Note: since metalog can answer head changes, I didn't add head changes of
commit cloud sync to the tracing data.
Reviewed By: DurhamG
Differential Revision: D19797700
fbshipit-source-id: b89924a7aa5e6027cad5c8138e8988f6b0ea4b2a
Summary: This allows us to query tracing data for watchman commands.
Reviewed By: DurhamG
Differential Revision: D19797711
fbshipit-source-id: 4dfd50fff820da70888faa0fe8f53af25f205137
