Summary:
Update hg.copystore and util.copyfiles to use the new-style progress bar
context manager.
Reviewed By: ryanmce
Differential Revision: D7329482
fbshipit-source-id: 4ac1def57e0ae4e7819c7c0c4d6f1715b8cbb625
Summary:
Previously, when looking for a treemanifest we'd inspect the flat
manifest revlog to get a hint for which nearby manifests might exist. This
assumed the data was in the flat manifest though, and if it wasn't it threw an
exception which caused us to not look at the server for the tree.
This commit fixes the case where the flat manifest doesn't have the commit, so
it can still look at the server.
Reviewed By: quark-zju
Differential Revision: D7343608
fbshipit-source-id: 57d5da9a8b8d71e33df9a8a0647b7bfe8815e37b
Summary:
Tests pushrebase can preserve commit hash if rewritedates is set to False.
This is the desired property mentioned in T26625986.
Reviewed By: DurhamG
Differential Revision: D7280498
fbshipit-source-id: e8943ddfec55a524c5cfd1dc9de765eace1e5296
Summary:
We have some repo's whose first commit didn't not add any files, which
means they have nullid as their manifest pointer which breaks backfilltrees.
This patch fixes that.
Reviewed By: singhsrb
Differential Revision: D7340678
fbshipit-source-id: 85b048b81c1861fd48d5cc082d414271aee7283b
Summary: Although the old glob *should* work, in practice the treemanifest matcher seems to have issues with glob:**/ paths, where relglob: works.
Reviewed By: ryanmce
Differential Revision: D7287992
fbshipit-source-id: d3802e730d937eeca7059dac691d8dac5a5de262
Summary:
This adds the ability to specify a config file to be used during the
command. This is useful during clones for letting the clone command use the
given repositories system specified repo-specific hgrc file.
Reviewed By: quark-zju
Differential Revision: D7311576
fbshipit-source-id: a97d8ebada2e0bea27c75a7650df8ede00dc10c6
Summary:
worddiff needs extra computation and does not have effect if color is
disabled. It should also be disabled with HGPLAIN=1.
Reviewed By: ryanmce
Differential Revision: D7315276
fbshipit-source-id: b59255bc708713186bd4ddafdc469763e1a66ea6
Summary:
There were recent complains about both quality [1] [2] and performance [3]
of the current word diff algorithm.
The current algorithm is actually bad in various ways:
- Lines could be matched across hunks, which is confusing (report [1]).
- For short lines, they can fail "similarity" check, which means they
won't be highlighted when they are expected to be (report [2]).
- Various performance issues:
- Using difflib implemented by pure Python, which is both slow and
suboptimal comparing with xdiff.
- Searching for matched lines across hunks could be O(N^2) if there are
no match found.
Thinking it in a "highlight" way is actually tricky, consider the following
change:
```
# before
foo = 10
# after
if True:
foo = 21 + 3
```
It's obvious that "10" and "21 + 3" need highlighting because they are
different. But what about "if True:"? In theory it's also "different" and
need highlighting. How about purely inserted or deleted hunks then?
Highlighting all of them would be too noisy.
This diff rewrites the word diff algorithm. It differs in multiple ways:
1. Get rid of "matching lines by similarity" step.
2. Only diff words within a same hunk.
3. Dim unchanged words. Instead of highlighting changed words.
4. Treat pure insertion or deletion hunks differently - do not dim or
highlight words in them.
5. Use xdiff instead.
6. Use a better regexp to split words. This reduces the number of tokens sent
to the diff algorithm.
1, 2, 5, 6 help performance. 1, 2, 3, 4 make the result more predictable and
trustworthy. 3 avoids the nasty question about what to highlight. 3 and 4 makes
it more flexible for people to tweak colors. 6 makes the result better since it
merges multiple space tokens into one so xdiff will less likely miss important
matches (than meaningless matches like spaces).
"bold" and "underline" were removed so the changed words will have regular
red/green colors. The output won't be too "noisy" even in cases where code are
changed in a way that inline word matching is meaningless. For people who want
more contrast, they can set:
[color]
diff.inserted.changed = green bold
diff.deleted.changed = red bold
Practically, when diffing D7319718, the old code spends 4 seconds on finding
matched lines preparing for worddiff:
```
| diffordiffstat cmdutil.py:1522
\ difflabel (17467 times) patch.py:2471
....
> 3927 \ _findmatches (22 times) patch.py:2537
348 \ __init__ (8158 times) difflib.py:154
340 | set_seqs (8158 times) difflib.py:223
328 | set_seq2 (8158 times) difflib.py:261
322 | __chain_b (8158 times) difflib.py:306
1818 \ ratio (8158 times) difflib.py:636
1777 | get_matching_blocks (8158 times) difflib.py:460
1605 \ find_longest_match (51966 times) difflib.py:350
38 | __new__ (51966 times) <string>:8
29 \ _make (36035 times) <string>:12
143 \ write (17466 times) ui.py:883
```
The new code takes 0.14 seconds:
```
| diffordiffstat cmdutil.py:1522
\ difflabel (23401 times) patch.py:2562
....
> 140 \ consumehunkbuffer (23346 times) patch.py:2585
130 | diffsinglehunkinline (23240 times) patch.py:2496
215 \ write (23400 times) ui.py:883
118 \ flush cmdutil.py:1606
118 | write ui.py:883
```
[1]: https://fburl.com/lkb9rc9m
[2]: https://fburl.com/0r9bqf0e
[3]: https://fburl.com/pxqznw31
Reviewed By: ryanmce
Differential Revision: D7314726
fbshipit-source-id: becd979cb9ac3fd3f4adae11cb10804d535f58df
Summary:
Instead of yielding tokens directly, buffer them if they belong to a same
hunk. This makes it easier for the upcoming new worddiff algorithm to only
focus on the diff hunk, instead of having to worry about other contents.
This breaks how the existing experimental worddiff algorithm works, so the
algorithm was removed, and related tests are disabled for now.
Reviewed By: ryanmce
Differential Revision: D7314725
fbshipit-source-id: 344e502cd185b2412fbd2ee299131bbb4560ac17
Summary:
A prior diff optimized push/pull to not send public trees between
peers, since those trees can be downloaded from the main server. Let's be
careful when sending data to the main server and always send everything.
In the future we should add validation on the server that the received data is
complete, but Mercurial doesn't currently do that today.
Differential Revision: D7296253
fbshipit-source-id: 49513685d19991a70d66da1d734ddae23491ed0c
Summary:
This allows pushing a treeonly pack to a server without using
pushrebase.
Differential Revision: D7295686
fbshipit-source-id: b0bfe4fbb04bc765e57f1db82909fa1ae7b3063b
Summary:
It can sometimes be nice to be able to just repack loose data and avoid existing packs.
This came up during building the megarepo, which creates many loose files, but including the packs added too much repack time.
Differential Revision: D7209296
fbshipit-source-id: 10afed40e409733e0ee004f025013cf86f3f7bf6
Summary: Help people debug issues by providing useful hints when templates fail to parse.
Differential Revision: D7148780
fbshipit-source-id: bd2b0f2bdeb0c970732fcaf53854d4a003b08571
Summary:
This logic is largely based on the similar logic added to template error
messages in D2608 and D2609, but with a few tweaks based on how revsets
actually work.
Differential Revision: D7148779
fbshipit-source-id: fb435788711a2c4ef881dfaeac5176fc1972c07a
Summary:
Previously we were just putting nullid as the linknode in client side
trees, because when the trees were added the changelog hadn't been written yet,
so we didn't know the linknode. This diff updates manifestlog.add to pass
linkrevs to mutablehistorypack, which get resolved at serialization time into
the appropriate linknode.
Reviewed By: ryanmce
Differential Revision: D7280104
fbshipit-source-id: bbc8a7bfad31da296a1b65973875ba2e1e1f7a95
Summary:
Previously we were just putting nullid as the linknode in client side
trees, because when the trees were added the changelog hadn't been written yet,
so we didn't know the linknode. This diff updates mutablehistorypack to allow
resolving the linknodes at serialization time instead of add time. This will be
used in a future diff to fix storing linknodes in trees.
Reviewed By: ryanmce
Differential Revision: D7280105
fbshipit-source-id: 70063e627d0fd7baeb017bac5ac55957a100d06c
Summary:
Previously converttotrees relied on being passed a linkrev to pass to
manifestlog.add(). If a linkrev wasn't provided though, it would pass None and
no linknode was stored. If a linknode is available, let's prefer that, and only
use the linkrev when we have to.
Reviewed By: ryanmce
Differential Revision: D7280109
fbshipit-source-id: d17015789a213fab6ff57563216e6943d2525fee
Summary:
Previously pushrebase pushes would send a tree pack part as well as a
rebase pack part. The tree pack part was ignored. Let's make receiving tree pack
parts on the server an error, and instead make the client not send them if it's
doing a pushrebase push.
A future diff will come back and add support for non-pushrebase push of tree
packs. This diff just makes use sure that the new logic won't run during
pushrebase.
Reviewed By: ryanmce
Differential Revision: D7267802
fbshipit-source-id: cabff989c178afb3706b77a03a299bcf3195f289
Summary:
Currently infinitepush relies on the ability to send flat manifests to
tree clients and the clients doing the conversion. This relies on a hack where
we don't pass the bundlecaps to the infinitepush changegroup creator, so it
doesn't realize it's talking to a treeonly client and therefore sends flat
manifests to a treeonly client. This is hacky. To fix this, let's start by
making it possible for the server to generate trees on demand from the flat
manifests. This will also be useful later when the server is treeonly, so it can
ondemand generate old flat bundles into a treeonly one.
A later diff will then pass the appropriate bundlecaps to the rebundle
changegroup function.
Reviewed By: ryanmce
Differential Revision: D7256554
fbshipit-source-id: 40d5d9537a5e11b3d186c8ec9a2579261f1b772e
Summary:
Add a new extension, `clienttelemetry`, which sends information about the
client to the server. This information is logged by the server, which can use
the `sampling` extension to store this information for later analysis.
Includes a few basic pieces of information by default, including a correlator
which can be used to correlate logs from the client and the server.
Reviewed By: mjpieters
Differential Revision: D7271508
fbshipit-source-id: f152fe177d8bc73179d3a2c4d0d070d468885e2a
Summary:
I somehow had a brain fart before landing my infinitepush phases diff
and replaced a hash with a rev number, even though rev numbers don't work with
infinitepush. This fixes that.
fbshipit-source-id: aa85614d40e173f6472cb9570af8510eb15ea353
Summary:
Add obsmarkers to the things that are synced.
For the LocalService, obsmarkers are stored base64-encoded using version 1 of
the obsmarker binary format. This is sufficient for unit tests.
Differential Revision: D7181399
fbshipit-source-id: 61377105986de561622a160134d20fdbd54bb88f
Summary:
Adds the initial skeleton of the `commitcloud` extension to Mercurial, which
will synchronise heads and bookmarks (and later, obsmarkers) with commit cloud.
Current limitations:
- only supports a stub version of the service that uses local files
- requires manual pushbackup to send commits to the server via infinitepush
- doesn't do anything with obsmarkers
Differential Revision: D7167715
fbshipit-source-id: 53443f3fb8dff0fe52257d7b20fc601dbe10c883
Summary: Put titles, if present, into a second 'column' in the plain output.
Differential Revision: D7258046
fbshipit-source-id: dedbb04ecbf0b2262c4c507683262e4b7bef8d70
Summary:
Profiles can now have a [metadata] section, with key-value pairs defining some metadata to be associated with the profile.
This supports multiline entries by starting a continuation line with some whitespace:
```
[metadata]
title = Some cool title
description = A longer description
will be joined with newlines
provided you start with a space
or a tab.
```
Differential Revision: D7257628
fbshipit-source-id: d6b09161d54cb6824f691d07caee83d0f8bd0a84
Summary: We regularly parse multiple profiles, through includes or when discovering. Be clear what profile file caused the issue.
Differential Revision: D7257636
fbshipit-source-id: 1203c13f6c6a4e4d1397ace818aee21b9db38550
Summary:
infinitepush manipulates what phases are sent back to the client. The
old behavior was that it would report every commit from the bundle to the client
as draft, even if the commit wasn't related to the heads being fetched and even
if the commit wasn't actually a root. The mercurial client logic expected these
values to actually be roots, and this caused some logic to act incorrectly and
mark commits unrelated to the pulled head public.
The fix is to only send roots, and only ancestors of the requested head.
Reviewed By: singhsrb
Differential Revision: D7265155
fbshipit-source-id: 709c3c807258049e44e493b0668f3034a0b11328
Summary:
There's a bug in infinitepush where it can cause client phases to get
screwed up. This affects pulls where multiple stacks were uploaded at once, then
only part of the stack is pulled. Let's add a test to expose the bug.
Reviewed By: singhsrb
Differential Revision: D7265104
fbshipit-source-id: ec1320d46ec865e8fbcf324e8d94737a5c111ef7
Summary: Delete an unnecessary test command which delete a non-existed local bookmark.
Reviewed By: StanislavGlebik
Differential Revision: D7216466
fbshipit-source-id: f8fc4215ef1fa725c5e5f92ef03cc2616326ac59
Summary:
Previously, if we weren't sending trees we would not attempt to process
what files needed to be sent. This was incorrect, since files may be sent
independent of what trees are decided to be sent. So let's update the code to
care about shouldaddfilegroup instead of cansendtrees.
This exposed an additional bug where we wouldn't look at the treemanifests
during the file computation, so when sending treeonly infinitepush bundles we
would get an error.
Reviewed By: singhsrb
Differential Revision: D7240340
fbshipit-source-id: a9be69597f4f1cbbecdd7cb1661f1023114bf621
Summary:
Sometimes infinitepush has to rebundle the data it's trying to send to
the client (like if you pull only part of a bundle). Previously, this was broken
for treeonly bundles because we did not correctly extend the bundlerepo's
data/history store on treemanifest servers. This patch fixes that and updates a
test.
Reviewed By: quark-zju
Differential Revision: D7240343
fbshipit-source-id: a9dd3ae884ace3fa9f4a748fe753fc394e69d6c9
Summary:
If we're pulling just part of an infinitepush bundle, it needs to
rebundle the data. We had a test for this for flat manifests, but we need a test
for treeonly commits as well. The test currently fails with an error. A future
diff will fix the error.
Reviewed By: quark-zju
Differential Revision: D7240342
fbshipit-source-id: 6acda3dfa6059f0957507a951df1b5e97df6dc4e
Summary:
Previously we just verified the pull worked. Let's also verify the
manifest is readable.
Reviewed By: quark-zju
Differential Revision: D7240341
fbshipit-source-id: 75c97591592699ff582fec5169ea20e0549fc603
Summary:
For short hashes it is nice to warn if the commit is too old.
Scenario:
After making commit on one machine, you can wait not enough time to
try to upload it on another machine using short hash from `hg sl`
in case of conflict (single conflict!) you can get automatically a very old commit,
that is not yours, without a warning. (And there were cases, when this screwed up hg that tried to go 250,000 commits ago)
I added a warning with a description of found commit if it is more than configured number of days old (default is 60)
Also, format of output has been redesigned. Now it is bold and got colors!
Also, I added #commitcloud
Another new features is that I added sorting by date in suggestions block.
Reviewed By: mjpieters
Differential Revision: D7195265
fbshipit-source-id: 7225a9b0fa8a7381f1e88ef7eb8a8fe2dfff43cd
Summary:
Once all clients are in treeonly mode, we can make the server tree only
mode as well and stop producing flat manifests entirely. This will speed up
pushes, save space, etc. This patch adds support for that mode.
Reviewed By: ryanmce
Differential Revision: D7148825
fbshipit-source-id: 15c899a58dd45af0c030c887c53f66347bc32b01
Summary:
The cstore doesn't contain our ondemand generator, nor our mutable data
store, so it's really not usable yet. Let's disable it for now until we can fix
up the issues that prevent us from using it (like having it report metrics).
Reviewed By: ryanmce
Differential Revision: D7148823
fbshipit-source-id: 5cc46af33c049b751c1c04916aafe0768d80ce7a
Summary:
Once we move to a treeonly world, there won't be any more manifest revlogs.
During the transition we'll still use hybrid manifest's though (the class, and
it will just resolve to treemanifests every time), so we need to support hybrid
manifests that don't actually have a revlog.
This also sets manifestlog._opener. We were already storing that on the other
manifestlog implementations, so we should do it on the main one for consistency
(and because this diff depends on it).
Reviewed By: ryanmce
Differential Revision: D7148821
fbshipit-source-id: 8f278a7bf4c0909423257165df59eae400a05a9e
Summary:
Previously the server would reject pushes that didn't use pushrebase,
since we relied on pushrebase to recreate the commits with the treemanifests.
Now that the flat-to-tree conversion logic is cleaner, we can use it to do the
conversion during a normal push.
Reviewed By: quark-zju
Differential Revision: D7143453
fbshipit-source-id: c13713e2ff59f1dff3ee1f44b7f8db7a92cfe1de
Summary:
This adds `progress.bar`, which is a context manager that provides a
time-updated progress bar, analogous to `progress.spinner`.
Reviewed By: ryanmce
Differential Revision: D7211895
fbshipit-source-id: be07d363f10c09164e2324d31a71a7c795257c8a
Summary:
Previously, in treeonly mode we would ignore any flat manifests that
were received in changegroups (via bundles or pulls, etc). This ended up causing
data loss in practice when people applied old bundles from before the
treemanifest conversion. Instead of just dropping those manifests, let's convert
them on the fly. This may be expensive, but it's better than losing the data.
A future diff may add caching to reuse flat text to speed up applying multiple
deltas.
Reviewed By: quark-zju
Differential Revision: D7083038
fbshipit-source-id: d2e350325d7e9005c8ddd5462034040274f790ff
Summary:
It makes sure that adding new options to the commands won't break them. For
example, rage sparse output was broken, and this diff fixes it.
Note that this changes behavior of the rage - if, say, smartlog extension is not enabled on the client, then there will be no output in the rage. Previously it wasn't the case. I think that's not the big problem, because:
a) The extensions are the most common ones and enabled everywhere
b) If they are disabled, then we'll immediately see a problem in hg rage.
However I need to change a test to make it pass. I need to add extensions and change grep output, because `hg rage --preview` has a few lines with `blackbox` in it.
Differential Revision: D7193297
fbshipit-source-id: dde2752ebc7dd3e3edea5c44576d0986f7d18744
Summary: sed -i without arguments doesn't work on OSX.
Reviewed By: farnz
Differential Revision: D7195193
fbshipit-source-id: a8eead927c94404a37ce5df956de82d29bc1b6a8