Summary:
I'm getting reports on Eden failed to import data from Mercurial. One of the error message I have seen is `local variable 'rcvd' referenced before assignment` (unfortunately we don't have the backtrace to locate the exact problem), and this is the only place in our codebase has a variable named `rcvd`.
The log contains this error: P62827869$210-217
So if there is any exception raised in `self._connect`, `rcvd` and `total` will not be initialized, and these variables are being used to log the error message in the `except` block:
diffusion/FBS/browse/master/fbcode/scm/hg/edenscm/hgext/remotefilelog/fileserverclient.py;18aa2052b752f1255dd53474d541c4a4177bfef5$732-737
Reviewed By: quark-zju
Differential Revision: D15069162
fbshipit-source-id: 16ec56820107fbbb24d426ce309b38a88d7eae5b
Summary: A more robust parsing regex for task references. It avoids some additional false positives, yet should cover all valid cases.
Reviewed By: Daij-Djan
Differential Revision: D15053374
fbshipit-source-id: 85410b5181eb9921d513b3a61ef3a1591b54539b
Summary:
The `path` argument passed to the follow revset is absolute. So `path` instead
of `relpath` should be used.
Reviewed By: DurhamG
Differential Revision: D15071189
fbshipit-source-id: 6aec76fa1a8cabd545a375aa40448cc75dbd1d6d
Summary: Release the GIL during data fetching to allow for progress bars to update properly. The data fetching code is pure Rust and does not interact with the Python interpreter at all, so releasing the GIL here is safe.
Differential Revision: D15051852
fbshipit-source-id: 144da953720951f9a30aadfc2b7fc8c8bc6b14aa
Summary:
When detecting split completion for the purposes of recording mutation, we used
`cmdutil.comparechunks`. This doesn't work for curses-recordings where the
only change is that some lines of a patch were excluded.
A simpler check is to just generate the patch that split is going to use, and
see if that matches the original patch.
Additionally, prevent the insertion of empty split records, as this can
cause crashes.
Reviewed By: mitrandir77
Differential Revision: D15063983
fbshipit-source-id: ba717d7f065faea93a500caaf10a1dbf582c7ab1
Summary:
Rather than relying on implicit conversion from a missing visibleheads file,
explicitly convert visibility information from obsmarkers when enabling
visibility tracking.
Reviewed By: mitrandir77
Differential Revision: D15063290
fbshipit-source-id: 44309f3cdf92c4ae100570b3bd98a240999ed558
Summary: Track visibility changes in the blackbox log to help with debugging issues.
Reviewed By: mitrandir77
Differential Revision: D15062971
fbshipit-source-id: c547618168f5eb08e6343e0b0d97db136e151a7a
Summary:
Ordinarily loops are prevented in the mutation graph as the predecessors must exist
at the point that the successor is created. However, backfilling from a complex
obsolescence graph may inadvertently introduce cycles.
Since loops are invalid, we can safely ignore any mutation edges that may
introduce them. The `allpredecessors` and `allsuccessors` functions already
do this.
Add loop detection and ignoring to the `predecessorsset` and `successorssets`
functions.
Reviewed By: mitrandir77
Differential Revision: D15062399
fbshipit-source-id: fe892d9236c8d8dc4e1322b82618ab4bca35d30a
Summary:
Use the graphnode `-` for all invisible commits, even obsolete ones.
Users will only see them in their logs if:
- they run log with `--hidden`.
- they have invisible commits that are temporarily unhidden (e.g. they've checked it out).
Reviewed By: mitrandir77
Differential Revision: D15061894
fbshipit-source-id: 86873bd86cb15cef72dae248b8e2a636378cc547
Summary:
When computing the fate of a commit, if we find the immediate successor is the
next visible commit, we return the operation of the mutation record for that
single operation. If it's not available, we then look at whether or not the
commit is public to decide between the fallbacks of `land` or `rewrite`.
Unfortunately, not all land operations end up with a `land` op. In particular,
obsmarkers created with pullcreatemarkers and then backfilled into mutation
records have a blank operation, and so we show `rewrite` here when we should
show `land`.
Switch around how we calculate the fate. Compute a default operation of `rewrite`
or `land`, based on the phase of the successor, and then use that if the successor
is not the immediate successor, or if the recorded operation is blank.
Reviewed By: mitrandir77
Differential Revision: D15061863
fbshipit-source-id: 753b0b58f84e653b40f9918f7ad3b3adfff359d8
Summary:
In the automigrate step at the start of pull, perform automigrations for
mutation and visibility.
If `mutation.automigrate` is set to true, then backfill obsmarkers into the
mutation store. This step can take a couple of seconds for large obsstores, so
print a message.
If `visibility.automigrate` is set to `"start"`, switch to explicit visibility
tracking. If it is set to `"stop"`, switch back to obsmarker-based tracking.
Reviewed By: mitrandir77
Differential Revision: D15046139
fbshipit-source-id: 284268d42b52c6b296c5c1b73db7bc218ae29a0a
Summary: Reading a comment is easier than trying to figure out the on-disk format.
Reviewed By: kulshrax
Differential Revision: D15056859
fbshipit-source-id: 097ed8bcaa51369aba4bcc9ed1cc95ebd6a67a66
Summary:
Compressing/Decompressing data can be expensive, so avoid doing it when not
needed. I though about using a RefCell but decided on just using mutable
reference as an Entry will always be private to indexedlogdatastore.rs.
Reviewed By: kulshrax
Differential Revision: D15056862
fbshipit-source-id: ac0b811f2df563be86e3ade9abe89476db5d13cc
Summary: This will allow decompression to be done on the fly as opposed to always.
Reviewed By: kulshrax
Differential Revision: D15056860
fbshipit-source-id: 60635c431579fc924a61d08b35688222ec4930bb
Summary:
Delta chains are only created during repack, as every download operation
fetches the full content of the file. Even if we wanted to support them,
interrupted chains adds undesirable complexity as it can lead to chain loops if
we're not careful. Let's just not support delta chains for now to avoid this.
Reviewed By: kulshrax
Differential Revision: D15056861
fbshipit-source-id: 4b0474ce134e946952a70f363190faf50850abe0
Summary: Now that IndexedLog are also in this crate, its name is no longer relevant.
Reviewed By: kulshrax
Differential Revision: D15056502
fbshipit-source-id: cb00c8322ac4ff7da97c8faaec2959e5f68ca4ca
Summary:
Once `remotefilelog.fetchpacks` is enabled, `hg gc` will no longer be able to
limit the size of the hgcache. This will be particularly challenging for
Sandcastle/Quicksand as they already see hgcache over 100GB.
The long-term plan is switching to IndexedLog based stores with a log rotate
functionality to control the cache size. In the meantime, we can implement
a basic logic to enforce the size of the hgcache that simply remove packfiles
once the cache is over the configured size.
One complication of this method is that several concurrent Mercurial processes
could be running and accessing the packfiles being removed. In this case, we
can split the packfiles in 2 categories: ones created a while back, and new
ones. Removing packfiles from the first case, lookups will simply raise a
KeyError and data will be re-fetched from Memcache/Mononoke, ie: failure is
acceptable. The second category belongs to ones that were just created by
downloading them from Memcache/Mononoke, and the code strongly assume that they
will stick around. A failure at this point will not be recovered.
One way of fixing this would be to handle these failures properly and simply
retry, the other is to not remove new packfiles. A time of 10 minutes was chosen
to categorize the packfiles.
Reviewed By: quark-zju
Differential Revision: D15014076
fbshipit-source-id: 014eea0251ea3a630aaaa75759cd492271a5c5cd
Summary:
Clean up some of the calls to `ui.log` and how they appear in blackbox logging.
* Make the names of the events consistently use `snake_case`.
* For watchman, only log once for each watchman command. Include whether or not it failed.
* Unify `fsmonitor` logging under the `fsmonitor` event.
* Omit the second argument when it is empty - it is optional and does nothing when empty.
* Increase the number of blackbox lines included in rage to 100.
Reviewed By: quark-zju
Differential Revision: D14949868
fbshipit-source-id: a9aa8251e71ae7ca556c08116f8f7c61ff472218
Summary: Per the title, if we attempt to fetch file data and history over HTTP and the fetch fails, fall back to SSH rather than crashing.
Differential Revision: D15035947
fbshipit-source-id: 2d00a49a51a0c8809daf1d28a6e3ab7f571415b0
Summary: Add a debug option for HTTP data fetching. The intended usage is for it to gate verbose debug messages; the option can be set by Chef for users in the hg_dev tier.
Differential Revision: D15040988
fbshipit-source-id: b7eaa3bab4200e083cffc5822fb9873611725e6b
Summary:
On Windows, all the tests that are expecting to find some files in $CACHEDIR
would fail due to the directory not existing. Interestingly enough, printing
$CACHEDIR would print a reasonable path, which is the same as $TESTTMP.
Trying to understand this better, I passed --keep-tmpdir to run-tests and
realized that the "real" $TESTTMP was somewhere in my home directory, while
the real $CACHEDIR was in fact C:\tmp.
I haven't fully understood why, but it looks like $PWD is expanded in C:\tmp,
while $TESTTMP is expanded into something else.
Reviewed By: quark-zju
Differential Revision: D15041274
fbshipit-source-id: 0d167183d74df5f6ab84360c5699e96808fceb9b
Summary: Almost all tests are failing on Windows due to a warning about BaseException.message being deprecated. Replace it with str(e).
Reviewed By: kulshrax
Differential Revision: D15039020
fbshipit-source-id: d984af91ec447b2f721eab2e3c6d39a0b350fb57
Summary:
The prechangegroup hook didn't have throw=True set, so if the hooks
failed we ignored it. This seems to have been the case for a long time, but we
only recently hit it.
Reviewed By: kulshrax
Differential Revision: D15038494
fbshipit-source-id: 4fa9ed4924c02732e3e4070e747a80fbe63564c9
Summary: Add a new config option to toggle file validation.
Differential Revision: D15034687
fbshipit-source-id: 3783ea1dacad9d1e494a5de1388f703db0ed1129
Summary: We have to pass a lot of config options across the FFI boundary; these are currently passed as arguments to the Eden API client constructor. Let's use argument unpacking to avoid repeating a bunch of argument names in the call to the constructor.
Differential Revision: D15034480
fbshipit-source-id: 74d0830c686c8863fcede6e57404aec3f0a58ea1
Summary:
When eden request a tree, it manually commit the pending mutable pack files. In
the unlikely case where the temporary files are removed from the disk, the
pack.close() operation will fail, since the pending packs aren't reset, the
next commit that happens while the repo object is closed will try again. This
time, it may try to close an already closed packfile, leading to P62634761.
Reviewed By: quark-zju
Differential Revision: D15015632
fbshipit-source-id: 016617334498c0161feed9dcec5ce24df931ad9c
Summary: Updated help text for hg amend and moved some options to verbose
Reviewed By: kulshrax
Differential Revision: D15004588
fbshipit-source-id: 4c9e0bffb522184ac8750ed8aa4eb5a53b309bd0
Summary:
I want to give Store a more specific name so that it doesn't get
confused with other Store abstractions that we will add in the
future.
Reviewed By: singhsrb
Differential Revision: D15007383
fbshipit-source-id: 499bcda4aecd5389e3bc1eba5206ba72a69c4c3d
Summary:
Python's `next()` can raise the `StopIteration` exception, unlike Rust's `next`
which just returns `None` instead.
Fix it by providing a default value `None`.
Reviewed By: singhsrb
Differential Revision: D15008773
fbshipit-source-id: df885c63b8130ceac38f86c89f2547dde2d519ba
Summary:
The warning isn't that useful, and can actually cause more harm than good, as running `hg prefetch -r .`
can download gigabytes of unnecessary data to the hgcache.
Reviewed By: quark-zju
Differential Revision: D14999458
fbshipit-source-id: b0ff2c2ad0e441622066fac10a5efafe8de588db
Summary:
The pdf generated by D14940136 used an older version of texlive. It has issues
with text colors after a footnotemark.
Correct it by recompiling with a newer version of texlive. Besides, I made some
minor edits to make the style more consistent (ex. capitalized some words,
removed some periods).
Reviewed By: sfilipco
Differential Revision: D15001655
fbshipit-source-id: 31bb7741ab18bba1594c553650e4710f537c2399
Summary:
`Log::lookup_range` exposes the range query feature provided by `Index`.
The iterator is made double-ended by the way.
Reviewed By: sfilipco
Differential Revision: D14895477
fbshipit-source-id: 6aef0973e009bf8fc6f3b5e5a8f6c54e57c81360
Summary:
The RangeIter is actually faster. The main reason is that it avoids recursion.
RangeIter does require double Vec, which seems like extra overhead. Practically
it does not seem to matter much.
The RangeIter code is also better written than PrefixIter. So let's delete
PrefixIter, and switch prefix lookups to use RangeIter.
Before:
index prefix scan (2B) 89.788 ms
index prefix scan (1B) 72.337 ms
index prefix scan (2B, disk) 102.098 ms
index prefix scan (1B, disk) 90.445 ms
After:
index prefix scan (2B) 76.335 ms
index prefix scan (1B) 54.517 ms
index prefix scan (2B, disk) 91.798 ms
index prefix scan (1B, disk) 67.143 ms
Reviewed By: sfilipco
Differential Revision: D14895478
fbshipit-source-id: 79a01774fb640c78fc5733db82f86f0f9403c960
Summary:
This would provide data about scan_prefix performance.
The benchmark code is slightly changed to share the index across test cases.
That reduces test setup cost.
Reviewed By: sfilipco
Differential Revision: D14895481
fbshipit-source-id: e70098bd202e102822a0829c0ae28de8d49fbe85
Summary:
This API allows range query, similar to `BTreeMap::range`.
It's going to be used by segmented changelog. There are spans (start, end)
stored in the index, and we need to find spans by rev (start <= rev <= end).
Initially I was changing PrefixIter incrementally towards the new RangeIter.
There are too many small commits and I got some useful feedback early. Now
it seems cleaner to just introduce the desired state of RangeIter first.
We can later migrate prefix lookup to RangeIter, if perf regression is
negligible.
The added code is long. But some of them are modified from existing code:
- `RangeIter::next_internal` is modified from `PrefixIter::next`.
- `Index::get_stack_by_bound` is modified from `Index::scan_prefix_base16`.
The tests helped find some issues of the code. I hope they're not too weak.
Reviewed By: sfilipco
Differential Revision: D14895479
fbshipit-source-id: fb8f1bd35c61187fe5f7764fa485206bbb13c8e0
Summary:
This test was broken by D14971701 on OSX because it has a case
insensitive filesystem.
Reviewed By: kulshrax
Differential Revision: D14986692
fbshipit-source-id: a2a924d7aae4f3b96e7691e824a82087c1ff8513
Summary:
The `if dest` test fails with infinitepush changing `dest` to non-None.
Fix it by also checking if `dest` matches `default-push` (or `default`).
Reported by: fryfrog
Reviewed By: markbt
Differential Revision: D14965995
fbshipit-source-id: 91e68368eda4457d06059387307a9572bc6d2906
Summary:
The way this command was implemented before this change would collapse
existing case collisions into a single value of the `lowerdirlist` dict.
The value that was chosen would be dependent on the traversal ordering.
If this value would be equivalent to the newly-introduced file, we would
miss the collision.
Reviewed By: markbt
Differential Revision: D14971701
fbshipit-source-id: d352e96e512154d92fe6bc49dd76aec63b954fef
Summary:
Change the fastlog approach to patch the "follow" revset function instead of
"getlogrevs". This makes it more general purposed so it can speed up commands
like `hg log -r 'reverse(follow("dir"))'`.
Note: This will regress `log dir` performance, which will be fixed in the next
diff.
Reviewed By: sfilipco
Differential Revision: D14764074
fbshipit-source-id: c2a4c8e91359d971e6ea668e5ff1f0ab6eb0534c
Summary:
The "hg log" implementation chooses to not use the "follow" revset when logging
a directory, but use "_matchfiles" instead. In an upcoming change, we'd like
"follow" to handle directories so fastlog only needs to patch the "follow"
revset.
The "follow" revset can take a directory pattern just fine. The problem is
"follow" will follow *every* file inside the specified directory, which is
quite expensive.
For now, I just moved the "_matchfiles" fallback path to "follow" so when it
detects there are too many files to follow, it will switch to "_matchfiles"
directly.
In theory, tree manfiest repos would have "tree manifest" infomation that speed
up "follow" on a directory. But that's a bigger change, and it's probably very
slow in our setup because our trees are lazy.
This changes some behaviors subtly, as reflected in tests:
- `-f path` can use DAG range instead of rev range, which is a good thing as
rev range does not make much sense to end-users. This removes a "BUG" in
test-commit-amend.t
- `-f dir` can follow renames if the directory contains just a few files.
This looks like a good thing, and is reflected in `test-log.t`.
Reviewed By: sfilipco
Differential Revision: D14863134
fbshipit-source-id: 99ddff46d43f63ce03dc7bf005e3ac1cb9b39d03
Summary:
With upcoming changes, I noticed `limit(reverse(:.) & follow("path:fbcode/scm/hg"), 10)`
is much slower than `limit(reverse(::.) & follow("path:fbcode/scm/hg"), 10)`. I tracked
it down to the fact that spanset introduces a lot of unnecessary checks. Optimize it
by avoid using spanset in this case.
The revset pattern is used by `hg log`, the `reverse(:.)` part is to "define" the
order. Perhaps we sould replace it with `reverse(::.)`. But that's a BC change that
might have some unwanted side effects.
Reviewed By: sfilipco
Differential Revision: D14863135
fbshipit-source-id: 6ba8a02b58e1109bdf8370f03965a3b302cba6c0
Summary: Update the Eden API client and server to use the new DataEntry type for file content downloads.
Reviewed By: quark-zju
Differential Revision: D14907958
fbshipit-source-id: 8a7b1cbb54bdc119dda11179ff94d3efdb7e85c9
Summary: Removing this function in favor of using Key.path
Reviewed By: quark-zju
Differential Revision: D14945331
fbshipit-source-id: 6b6bb70375629edf37b2b04a86545f18e15b33b4
