Summary:
Update hg.copystore and util.copyfiles to use the new-style progress bar
context manager.
Reviewed By: ryanmce
Differential Revision: D7329482
fbshipit-source-id: 4ac1def57e0ae4e7819c7c0c4d6f1715b8cbb625
Summary:
New-style progress bars should only update if there has been any progress since
the last refresh. Otherwise, stalls in progress cause weird behaviour for time
estimates.
Reviewed By: DurhamG, quark-zju
Differential Revision: D7329494
fbshipit-source-id: 43b9fdd5573b34174673db838e53a0d54504eefd
Summary:
`progress.debugbar` is an implementation of the `progress.bar` context that
behaves like the old progress bar implementation when used in tests.
Reviewed By: DurhamG
Differential Revision: D7329500
fbshipit-source-id: 28c96c1ddf096efbec6994cf5b4dea04f88d55d3
Summary:
We have some repo's whose first commit didn't not add any files, which
means they have nullid as their manifest pointer which breaks backfilltrees.
This patch fixes that.
Reviewed By: singhsrb
Differential Revision: D7340678
fbshipit-source-id: 85b048b81c1861fd48d5cc082d414271aee7283b
Summary:
This adds the ability to specify a config file to be used during the
command. This is useful during clones for letting the clone command use the
given repositories system specified repo-specific hgrc file.
Reviewed By: quark-zju
Differential Revision: D7311576
fbshipit-source-id: a97d8ebada2e0bea27c75a7650df8ede00dc10c6
Summary:
worddiff needs extra computation and does not have effect if color is
disabled. It should also be disabled with HGPLAIN=1.
Reviewed By: ryanmce
Differential Revision: D7315276
fbshipit-source-id: b59255bc708713186bd4ddafdc469763e1a66ea6
Summary:
There were recent complains about both quality [1] [2] and performance [3]
of the current word diff algorithm.
The current algorithm is actually bad in various ways:
- Lines could be matched across hunks, which is confusing (report [1]).
- For short lines, they can fail "similarity" check, which means they
won't be highlighted when they are expected to be (report [2]).
- Various performance issues:
- Using difflib implemented by pure Python, which is both slow and
suboptimal comparing with xdiff.
- Searching for matched lines across hunks could be O(N^2) if there are
no match found.
Thinking it in a "highlight" way is actually tricky, consider the following
change:
```
# before
foo = 10
# after
if True:
foo = 21 + 3
```
It's obvious that "10" and "21 + 3" need highlighting because they are
different. But what about "if True:"? In theory it's also "different" and
need highlighting. How about purely inserted or deleted hunks then?
Highlighting all of them would be too noisy.
This diff rewrites the word diff algorithm. It differs in multiple ways:
1. Get rid of "matching lines by similarity" step.
2. Only diff words within a same hunk.
3. Dim unchanged words. Instead of highlighting changed words.
4. Treat pure insertion or deletion hunks differently - do not dim or
highlight words in them.
5. Use xdiff instead.
6. Use a better regexp to split words. This reduces the number of tokens sent
to the diff algorithm.
1, 2, 5, 6 help performance. 1, 2, 3, 4 make the result more predictable and
trustworthy. 3 avoids the nasty question about what to highlight. 3 and 4 makes
it more flexible for people to tweak colors. 6 makes the result better since it
merges multiple space tokens into one so xdiff will less likely miss important
matches (than meaningless matches like spaces).
"bold" and "underline" were removed so the changed words will have regular
red/green colors. The output won't be too "noisy" even in cases where code are
changed in a way that inline word matching is meaningless. For people who want
more contrast, they can set:
[color]
diff.inserted.changed = green bold
diff.deleted.changed = red bold
Practically, when diffing D7319718, the old code spends 4 seconds on finding
matched lines preparing for worddiff:
```
| diffordiffstat cmdutil.py:1522
\ difflabel (17467 times) patch.py:2471
....
> 3927 \ _findmatches (22 times) patch.py:2537
348 \ __init__ (8158 times) difflib.py:154
340 | set_seqs (8158 times) difflib.py:223
328 | set_seq2 (8158 times) difflib.py:261
322 | __chain_b (8158 times) difflib.py:306
1818 \ ratio (8158 times) difflib.py:636
1777 | get_matching_blocks (8158 times) difflib.py:460
1605 \ find_longest_match (51966 times) difflib.py:350
38 | __new__ (51966 times) <string>:8
29 \ _make (36035 times) <string>:12
143 \ write (17466 times) ui.py:883
```
The new code takes 0.14 seconds:
```
| diffordiffstat cmdutil.py:1522
\ difflabel (23401 times) patch.py:2562
....
> 140 \ consumehunkbuffer (23346 times) patch.py:2585
130 | diffsinglehunkinline (23240 times) patch.py:2496
215 \ write (23400 times) ui.py:883
118 \ flush cmdutil.py:1606
118 | write ui.py:883
```
[1]: https://fburl.com/lkb9rc9m
[2]: https://fburl.com/0r9bqf0e
[3]: https://fburl.com/pxqznw31
Reviewed By: ryanmce
Differential Revision: D7314726
fbshipit-source-id: becd979cb9ac3fd3f4adae11cb10804d535f58df
Summary:
Instead of yielding tokens directly, buffer them if they belong to a same
hunk. This makes it easier for the upcoming new worddiff algorithm to only
focus on the diff hunk, instead of having to worry about other contents.
This breaks how the existing experimental worddiff algorithm works, so the
algorithm was removed, and related tests are disabled for now.
Reviewed By: ryanmce
Differential Revision: D7314725
fbshipit-source-id: 344e502cd185b2412fbd2ee299131bbb4560ac17
Summary:
The original logic makes it harder to reason about - it yields the "\n"
character belonging to the last line in the next loop iteration.
The new code is in theory a little bit slower. But is more readable. It
makes the following changes easier to read.
Reviewed By: ryanmce
Differential Revision: D7314727
fbshipit-source-id: cb792577adc8f7629bc2e0fe8f0688e705bf4b30
Summary:
Previously, during a push the changegroup packer had no information about the
destination server, aside from what changegroup version it supported. In a
future diff we'll want to package certain files and trees based on if the server
is a main server or not, so let's pass thread the bundle2caps down to the
changegroup packer.
The changegroup already had the old style bundlecaps on it, but those
capabilities don't actually exist on the client side during a push, so let's
just use the bundle2 ones, which are more modern anyway.
Differential Revision: D7297899
fbshipit-source-id: 74ab529baccbdc999267555779a10c0d27f8bd47
Summary: Help people debug issues by providing useful hints when templates fail to parse.
Differential Revision: D7148780
fbshipit-source-id: bd2b0f2bdeb0c970732fcaf53854d4a003b08571
Summary:
This logic is largely based on the similar logic added to template error
messages in D2608 and D2609, but with a few tweaks based on how revsets
actually work.
Differential Revision: D7148779
fbshipit-source-id: fb435788711a2c4ef881dfaeac5176fc1972c07a
Summary:
Previously we were just putting nullid as the linknode in client side
trees, because when the trees were added the changelog hadn't been written yet,
so we didn't know the linknode. This diff updates manifestlog.add to pass
linkrevs to mutablehistorypack, which get resolved at serialization time into
the appropriate linknode.
Reviewed By: ryanmce
Differential Revision: D7280104
fbshipit-source-id: bbc8a7bfad31da296a1b65973875ba2e1e1f7a95
Summary:
In a future diff we'll want to allow converting flat manifests to tree
manifests during infinitepush. To do so, let's allow writing to in memory packs
on bundlerepositories.
Reviewed By: StanislavGlebik
Differential Revision: D7256563
fbshipit-source-id: 10ec58d1171b7882d6db9a916c50a19bc11dbcb4
Summary:
Once all clients are in treeonly mode, we can make the server tree only
mode as well and stop producing flat manifests entirely. This will speed up
pushes, save space, etc. This patch adds support for that mode.
Reviewed By: ryanmce
Differential Revision: D7148825
fbshipit-source-id: 15c899a58dd45af0c030c887c53f66347bc32b01
Summary:
Once we move to a treeonly world, there won't be any more manifest revlogs.
During the transition we'll still use hybrid manifest's though (the class, and
it will just resolve to treemanifests every time), so we need to support hybrid
manifests that don't actually have a revlog.
This also sets manifestlog._opener. We were already storing that on the other
manifestlog implementations, so we should do it on the main one for consistency
(and because this diff depends on it).
Reviewed By: ryanmce
Differential Revision: D7148821
fbshipit-source-id: 8f278a7bf4c0909423257165df59eae400a05a9e
Summary:
Previously the server would reject pushes that didn't use pushrebase,
since we relied on pushrebase to recreate the commits with the treemanifests.
Now that the flat-to-tree conversion logic is cleaner, we can use it to do the
conversion during a normal push.
Reviewed By: quark-zju
Differential Revision: D7143453
fbshipit-source-id: c13713e2ff59f1dff3ee1f44b7f8db7a92cfe1de
Summary:
This adds `progress.bar`, which is a context manager that provides a
time-updated progress bar, analogous to `progress.spinner`.
Reviewed By: ryanmce
Differential Revision: D7211895
fbshipit-source-id: be07d363f10c09164e2324d31a71a7c795257c8a
Summary:
The normal `ui.progress` function is noticably slow due to the debug code, even
when debug is disable. Prevent this from happening by bypassing it completely
when debug is not enabled.
Reviewed By: ryanmce
Differential Revision: D7213855
fbshipit-source-id: ecea0e610ba01ad35c44cd7bd8cac74b8e00621d
Summary:
During profiling of the `hg sparse --refresh` command, I discovered
significant time spent in producing the progress bar. At first I was skeptical,
but then I commented out the progress line and the command took half as long!
I started looking for micro-optiization opportunities.
Differential Revision: D7210617
fbshipit-source-id: 17a11ae572fee48e6d8db33ca75843be0ec2e850
Summary:
Add a function that returns command and all the default options already
initialized. It should be used by commands that call other commands. For
example, calling pull inside of update, calling log inside of show etc.
getcmdanddefaultopts has important benefits:
1) It returns "wrapped" command i.e. command with all the overrides applied. On
the other hand, commands.pull doesn't return it.
2) It correctly initializes options to their default value and correctly
changes their name - replace '-' with '_'.
Reviewed By: ryanmce
Differential Revision: D7193296
fbshipit-source-id: e8673bd4e16aad6156498660f2a7ed788ed2cac3
Summary:
This both fixes semantics and makes `compat.h` a bit more readable.
This because necessary, because we migtrated from external `compat.h` to `mercurial/compat.h` in D7064623.
Reviewed By: DurhamG
Differential Revision: D7196403
fbshipit-source-id: 0005cc2f4e58951adfe8f7f795067da728ad64ae
Summary:
In D7001328 we've added a new feature that skips commtis if there are no
changes relative to the sparse checkout. Unfortunately that causes lots of
treepacks downloads and makes bisect unusable. Let's revert the change.
Reviewed By: ryanmce, farnz
Differential Revision: D7182016
fbshipit-source-id: 274b29ca6a7b4c3faf83883b64f5ad3b0289873e
Summary:
Rewriting a set of commits where there are replacement relationship among the
commits do not have an optimal UX today. For example, `rebase -s A -d Z` or
`metaedit A` in the below graph. B1, B2, C will all be replaced. But the new B1
and B2 replacement won't have the B1 -> B2 relationship, and the "new B1"
appears to be revived.
```
o C
|
x B1 (amended as B2)
|
| o B2
|/
o A o Z
```
One solution is to avoid rebasing `obsolete()::`, as implemented in D7067121
for metaedit. That would result in
```
o C
|
x B1 (amended as B2) o new B2
| |
x A o new A
```
The stack of A, B1, C is forced to break into two parts. This is fine for
power users. But n00b users would wonder why C is left behind. Per discussion
with simpkins at an internal post about the metaedit case, we think a more
linear history is more user-friendly. That is:
```
o new C
|
x new B1 (amended as *new* B2)
|
| o new B2
|/
o new A
```
The stack stays in a same shape.
This diff implements the "copying obsmarkers" behavior at the "createmarkers"
level so everything using that API would get the feature for free, including
metaedit and rebase.
D7067121 is reverted since the new UX is preferred. The test added is for
`metaedit` command, changes to rebase will be added in a later patch.
Differential Revision: D7121487
fbshipit-source-id: fd3c8a96ab434b131fb86d9882ccbdff8f63f05e
Summary:
This diff adds a config option to tweak deltabase in changegroup. It has 3
options:
- Always null - always use "null" as delta base, effectively make
everything full text
- No external - delta bases cannot be a revision outside the changegroup
- Default - the current behavior: delta bases can be anything that client
thinks the server should have.
This gives Mononoke more time to bake delta related logic, as we can
choose "always null" first, then incrementally increase the complexity.
Reviewed By: phillco
Differential Revision: D7158585
fbshipit-source-id: 5f6d9a78d1108093e8d08b9f296568f4f7e7471b
Summary:
Verify had some logic that checked the length of the changelog and
manifest to decide if either existed. This allowed for simplifying certain error
messages (like not reporting all the broken changelog manifest pointers if the
manifest was simply gone, and just reporting the manifest was gone).
Unfortunately, in future changelog and manifest implementations len() will be an
expensive function, so let's just get rid of that optimization.
This fixes hg verify in a treeonly repository.
Reviewed By: quark-zju
Differential Revision: D7127168
fbshipit-source-id: 8ddc3dfe3c3c913efd4b7af5fc9715a3e48b60a1
Summary:
For large updates, the dirstate update can take a while. Let's show
progress so the user understands what is happening and how long to wait.
Reviewed By: quark-zju
Differential Revision: D6903133
fbshipit-source-id: f7f6c3c14e1d3221a383da4a6e311aa12a8d3a98
Summary:
Let's switch to xdiff for its better diff quality and performance!
The test changes demonstrate xdiff's better diff quality.
Reviewed By: ryanmce
Differential Revision: D7135206
fbshipit-source-id: 1775df6fc0f763df074b4f52779835d6ef0f3a4e
Summary:
Enable the indent heuristic feature, since it provides nice visual
improvements for a wide range of cases. See the added test, and [1].
The only downside is it can slow things down. In a crafted case, this could
make `--indent-heuristic` several times slower than `--no-indent-heuristic`.
```
open('a', 'w').write(" \n" * 1000000)
open('b', 'w').write(" \n" * 1000001)
```
```
git diff --no-indent-heuristic a b 0.21s user 0.03s system 100% cpu 0.239 total
git diff --indent-heuristic a b 0.77s user 0.02s system 99% cpu 0.785 total
```
[1]: 433860f3d0
Reviewed By: ryanmce
Differential Revision: D7135452
fbshipit-source-id: 019b7e89225f288bba0a1d042591b13b5419ad0e
Summary:
Implement a `mercurial.cext.xdiff` module that exposes the xdiff algorithm.
`xdiff.blocks` should be a drop-in replacement for `bdiff.blocks`.
In theory we can change the pure C version of `bdiff.c` directly. However
that means we lose bdiff entirely. It seems more flexible to have both at
the same time so they can be easily switched via Python code. Hence the
Python module approach.
Reviewed By: ryanmce
Differential Revision: D7135205
fbshipit-source-id: 48cd3b5be7fd5ef41b64eab6c76a5c8a6ce99e05
Summary:
verify.skipmanifests was added to let us skip the parts of verify that
were expensive in a lazy-manifest world. We also need to skip .hgsubstate
verification since it requires the manifests.
Reviewed By: singhsrb
Differential Revision: D7127353
fbshipit-source-id: 377fffb8556f7578da3e51c3da53f97554fb5d74
Summary: This change minimally addresses the issue that `debugrebuilddirstate` can crash if the dirstate file is very corrupt.
Reviewed By: markbt
Differential Revision: D7028370
fbshipit-source-id: 72fc7a2900a8bc1bb5f062454530b4fc4c806f09
Summary:
This logic exists for the following case: if becuase of some circumstances (user error, NFS/Samba weirdness) a symlink file becomes corrupted and we check this file in while preserving the 'l' flag in the manifest, the revision becomes un-checoutable on Posix filesystems (according to the original explanations anyway).
This option basically adds an ability to take this risk. I am planning to enable this option for `ovrsource`, since it already has very few symlinks so the probability of corrupting something is pretty low.
Differential Revision: D7127333
fbshipit-source-id: f85d9f3aef676afca641280c9b7f0ecfb87b9fab
Summary: dispatch.lazyaliasentry didn't work with slices i.e. lazyasliasentry[:2] failed. This diff fixes it
Reviewed By: farnz
Differential Revision: D7109779
fbshipit-source-id: c704cd44fea0944ae4be68df36d32df98b7fc09b
