Commit Graph

19530 Commits

Author SHA1 Message Date
Augie Fackler
8585259ff1 exchange: use '%d' % x instead of str(x) to encode ints
Recommended by Yuya instead of using pycompat.bytestr() in this case.
2017-08-22 21:21:13 -04:00
Augie Fackler
d68f01169b posix: always pass a native str to unicodedata.normalize's first arg 2017-09-15 19:44:32 -04:00
Augie Fackler
e4d2f96bbd posix: use slicing to grab a single byte out of a bytes in HFS+ normcase code 2017-09-15 19:44:05 -04:00
Augie Fackler
7fd201a134 encoding: ensure getutf8char always returns a bytestr, never an int 2017-09-15 19:43:32 -04:00
Augie Fackler
e6768c401f posix: fix HFS+ normcase doctest to produce valid bytes literals in Python 3
We were previously getting lucky on Python 2.
2017-09-15 19:43:02 -04:00
Gregory Szorc
f95c9311db show: pass the minimum length for nodes as a template keyword
This will allow us to make the displayed length configurable
and/or dynamic.

Differential Revision: https://phab.mercurial-scm.org/D556
2017-08-03 21:51:34 -07:00
Gregory Szorc
c904fe4a55 cmdutil: allow extra properties to be added to each context
The changeset displayer allows setting extra keywords to be available
to the templating layer. This patch adds an argument to displaygraph()
to pass a dict of extra properties to be available to every changeset.

Differential Revision: https://phab.mercurial-scm.org/D555
2017-08-03 21:13:27 -07:00
Michael Bolin
454c5f00c6 dirstate: perform transactions with _map using single call, where possible
This is in the same style as https://phab.mercurial-scm.org/D493.

In general, this replaces patterns such as:

```
f in self._map:
    entry = self._map[f]
```

with:

```
entry = self._map.get(f):
if entry is not None:
    # use entry
```

Test Plan:
`make tests`

Differential Revision: https://phab.mercurial-scm.org/D663
2017-09-14 09:41:22 -07:00
Boris Feld
80d3295191 extensions: register config item early
Config items are likely to be used in during extensions setup. So we much
register them before that.

For example this apply to the 'win32text.warn' options.
2017-09-05 00:34:13 +02:00
Boris Feld
8744970b58 extensions: factor extra data loading out
Some of the extra data need to be registered earlier than they currently are
(eg: config items). We first factor out the logic to registered them in a small
function before reusing it in the next changeset.
2017-09-05 00:31:59 +02:00
Pulkit Goyal
a5baff1381 copytrace: move fast heuristic copytracing algorithm to core
copytrace extension in fb-hgext has a heuristic implementation of copy tracing
which is faster than the current copy tracing. The heuristic limits the search
of copies to just files that are either:

1) Renames in the same directory
2) Moved to other directory with same name

The default copytrace implementation is very slow as it finds all the new files
that were added from merge base up to the head commit and for each file it
checks whether it this was copied or moved version of a different file.

Stash@fb did analysis for the above heuristics on the fb repo and found that
among 2,443,768 moves/copies there are only 32,234 moves/copies which does not
fall under the above heuristics which is approx. 0.013 of total copies.

This patch moves the heuristics algorithm under config
`experimental.copytrace=heuristics`.

While moving fbext to core, this patch removes couple of less useful config
options named `sourcecommitlimit` and `maxmovescandidatestocheck`.

Tests are also added for the heuristics algorithm, which are basically copied
from fbext/tests/test-copytrace.t. The tests follow a pattern creating a server
repo and then cloning to a local repo to create public and draft changesets, the
distinction which will be useful in upcoming patches.

After this patch `experimental.copytrace` has the following behaviour:

1) `off`: turns off copytracing
2) `heuristics`: use the heuristic algorithm added in this patch.
3) everything else: use the full copytracing algorithm

.. feature::

   A new fast heuristic algorithm for copytracing which assumes that the files
   moves are either::
   1) Renames in the same directory
   2) Moves in other directories with same names
   You can use this algorithm by setting `experimental.copytrace=heuristics`.

Differential Revision: https://phab.mercurial-scm.org/D623
2017-09-03 03:49:15 +05:30
Durham Goode
f57a683d4b bundle2: move exception handling into part iterator
As part of separating the part iteration logic from the part handling logic,
let's move the exception handling to the part iterator class.

Differential Revision: https://phab.mercurial-scm.org/D705
2017-09-13 20:39:01 -07:00
Durham Goode
a46fac03a2 bundle2: move part counter to partiterator
As part of moving the part iterator logic to a separate class, let's move the
part counting logic and the output for it.

Differential Revision: https://phab.mercurial-scm.org/D704
2017-09-13 17:16:50 -07:00
Durham Goode
577e9d1779 bundle2: move part iterator a separate class
Currently, the part iterator logic is tightly coupled with the part handling
logic, which means it's hard to replace the part handling logic without
duplicating the part iterator bits.

In a future diff we'll want to be able to replace all part handling, so let's
begin refactoring the part iterator logic to it's own class.

Differential Revision: https://phab.mercurial-scm.org/D703
2017-09-13 17:16:45 -07:00
Durham Goode
b8aaeea8b3 changegroup: add source parameter to generatemanifests
Extensions, like remotefilelog, will want to look at the source of a pull when
determining what manifests to add to a changegroup. For instance, on push they
will include everything, while on pull they won't.

Differential Revision: https://phab.mercurial-scm.org/D686
2017-09-11 13:39:22 -07:00
Durham Goode
c94bd0e75c changegroup: remove changegroup dependency from revlog.addgroup
Previously revlog.addgroup would accept a changegroup and a linkmapper and use
it to iterate of the deltas. As part of untangling the revlog-changegroup
interdependency, let's move the changegroup delta iteration logic to it's own
function and pass the simple iterator to the revlog instead.

This will make it easier to introduce non-revlogs stores in the future, without
reinventing any changegroup specific logic.

Differential Revision: https://phab.mercurial-scm.org/D688
2017-09-13 10:43:44 -07:00
Durham Goode
d27ceccf8a revlog: refactor chain variable
Previously the addgroup loop would set chain to be the result of
self._addrevision(node,...). Since _addrevision now always returns the passed in
node, we can drop that behavior and just always set chain = node in the loop.

This will be useful in a future patch where we refactor the cg.deltachunk logic
to another function and therefore chain disappears entirely from this function.

Differential Revision: https://phab.mercurial-scm.org/D699
2017-09-13 10:43:16 -07:00
Mark Thomas
3dfe8fba1c scmutil: don't append .orig to backups in origbackuppath (BC)
When ui.origbackuppath is set, .orig files are stored outside of the working
copy, however they still have a .orig suffix appended to them.  This can cause
unexpected conflicts, particularly when tracked files or directories have .orig
at the end.

This change removes the .orig suffix from files stored in an out-of-tree
origbackuppath.

Test Plan:
Update and run unit tests.

Differential Revision: https://phab.mercurial-scm.org/D679
2017-09-11 17:49:49 +00:00
Phil Cohen
be35a1685b merge: move cwd-missing detection to helper functions
This will exist in two places with defered writes, so we want to avoid
duplication.

Differential Revision: https://phab.mercurial-scm.org/D626
2017-09-12 19:27:01 -07:00
Yuya Nishihara
69001a6da3 doctest: coerce dict.keys() to list
Otherwise it would be printed as odict_keys([...]) on Python 3.
2017-09-03 17:33:10 +09:00
Yuya Nishihara
6de6cb029d doctest: upgrade old-style "except" clause 2017-09-03 15:16:01 +09:00
Yuya Nishihara
dcc07e5503 doctest: use print_function and convert bytes to unicode where needed 2017-09-03 14:56:31 +09:00
Yuya Nishihara
21687d1395 doctest: do not embed non-ascii characters in docstring
Since the outer docstring is parsed as a unicode on Python 3, we have to
either double-escape or construct non-ascii string from ascii string.
2017-09-03 15:47:17 +09:00
Yuya Nishihara
c7d6fe4d91 doctest: pass encoding name as system string 2017-09-03 15:42:27 +09:00
Yuya Nishihara
a20b4175e2 doctest: replace str() with bytes() 2017-09-03 14:38:58 +09:00
Yuya Nishihara
0435ef1b07 doctest: replace chr() with pycompat.bytechr() 2017-09-03 14:37:25 +09:00
Yuya Nishihara
b8768dba29 doctest: replace .iteritems() with .items() 2017-09-03 14:35:37 +09:00
Yuya Nishihara
a71f259bd2 doctest: bulk-replace string literals with b'' for Python 3
Our code transformer can't rewrite string literals in docstrings, and I
don't want to make the transformer more complex.
2017-09-03 14:32:11 +09:00
Yuya Nishihara
e116353ac2 debuginstall: do not pass exception object to formatter (issue5676) 2017-09-07 22:36:54 +09:00
Yuya Nishihara
c3d677e9b5 debuginstall: use codecs.lookup() to detect invalid encoding
encoding.fromlocal() never tries to decode an ascii string since 3cb2361c60fc,
and there's no universal non-ascii string which can be decoded as any valid
character set.
2017-09-07 22:27:23 +09:00
Yuya Nishihara
04508d7f1c extensions: fix wrapcommand/function of class instance
5f4c097a17d2 changed _updatewrapper() to copy the __name__ attribute, but
not all callable objects has __name__.

Spotted by loading mq with extdiff.
2017-09-10 23:37:14 +09:00
Durham Goode
792f16d38c changegroup: avoid creating empty changegroup part
Previously this check happened in the changegroup code itself. Since its
refactor, this logic needs to move out to callers that care about it, such as
this one. Otherwise we get empty bundle devel-warnings in certain extensions.

Differential Revision: https://phab.mercurial-scm.org/D690
2017-09-12 09:13:02 -07:00
Phil Cohen
b481a15354 merge: flush any deferred writes just before recordupdates()
``recordupdates`` calls into the dirstate which requires the files to be
there, so this is the last possible moment we can flush anything.

Differential Revision: https://phab.mercurial-scm.org/D673
2017-09-11 13:17:43 -07:00
Phil Cohen
0e9cb373ec merge: flush any deferred writes before, and after, running any workers
Since we fork to create workers, any changes they queue up will be lost after
the worker terminates, so the easiest solution is to have each worker flush
the writes they accumulate--we are close to the end of the merge in any case.

To prevent duplicated writes, we also have the master processs flush before
forking.

In an in-memory merge (M2), we'll instead disable the use of workers.

Differential Revision: https://phab.mercurial-scm.org/D628
2017-09-11 13:03:27 -07:00
Phil Cohen
152d58b25d filemerge: flush if using deferred writes when running a merge tool
Since merge tools might read from the filesystem, we need to write out our
deferred writes here.

No-ops if not using deferred writes.

Differential Revision: https://phab.mercurial-scm.org/D627
2017-09-11 13:03:27 -07:00
Phil Cohen
a1eba8f292 merge: pass wctx to premerge, filemerge
In the in-memory merge branch. we'll need to call a function (``flushall``) on
the wctx inside of _xmerge.

This prepares the way so it can be done without hacks like ``fcd.ctx()``.

Differential Revision: https://phab.mercurial-scm.org/D449
2017-09-11 13:03:27 -07:00
Boris Feld
60d92de7ba cmdutil: fix amend when passing a date
Following https://phab.mercurial-scm.org/D636, passing the same date that the
changeset to amend would results in no new commits but the output changed
from:

  $ hg amend -d '0 0'
  nothing changed
  [1]

to:

  $ hg amend -d '0 0'

Restore the old behavior by parsing the date passed as parameter so the
condition "date == old.date()" correctly works in cases both dates are
identical.

Add a test for covering this regression.

This bug was found thanks to Evolve test suite.

Differential Revision: https://phab.mercurial-scm.org/D691
2017-09-11 18:07:29 +02:00
Augie Fackler
85e2672134 merge with stable 2017-09-12 11:18:35 -04:00
Matt Harbison
19d34e4d32 debugssl: allow a URL to be specified without a local repository
This was the original intent, but I bungled the logic.  Otherwise if there is a
certificate chain issue, the repository can't be cloned in order for there to be
a repo object.  I think I missed this case because I was inside of a Mercurial
clone as I was originally developing and testing this.
2017-08-29 16:38:10 -04:00
Jun Wu
45a4782018 record: fix revert -i for lines without newline (issue5651)
This is a regression caused by 10c1efcbeb1e. Code prior to 10c1efcbeb1e
seems to miss the "\ No newline at end of file" line.

Differential Revision: https://phab.mercurial-scm.org/D528
2017-08-27 13:39:17 -07:00
Martin von Zweigbergk
7cd70adbc1 templatekw: choose {latesttag} by len(changes), not date (issue5659)
As Augie reported in the bug, the current heuristic of choosing the
best tag of a merge commit by taking the one with newest tag (in terms
of tagging date) currently fails in the Mercurial repo itself. Copying
the example from Yuya:

  $ hg glog -T '{node|short} {latesttag}+{latesttagdistance}\n' \
    -r '4.2.3: & (merge() + parents(merge()) + tag())'
  o    cc59efae4cc0 4.2.3+5
  |\
  | o    06f60e88fc3a 4.2.3+4
  | |\
  | | o  c191a9eb0b10 4.3-rc+109
  | | |
  | | ~
  o |  49ada93fdc10 4.3.1+2
  : |
  o |  229937197835 4.3.1+0
  |/
  o    6a83ad94c0f2 4.2.3+3
  |\
  | ~
  o  8e9dcdd1de74 4.2.3+2
  :
  o  525f2b18248f 4.2.3+0
  |
  ~

It seems to me like the best choice is the tag with the smallest
number of changes since it (across all paths, not the longest single
path). So that's what this patch does, even though it's
costly. Best-of-5 timings for Yuya's command above shows a slowdown
from 1.293s to 1.610s. We can optimize it later.

Differential Revision: https://phab.mercurial-scm.org/D447
2017-08-15 23:23:55 -07:00
Gregory Szorc
8e7a19b422 ui: restore behavior to ignore some I/O errors (issue5658)
45345e9870c3 and b30126fa95bc refactored ui methods to no longer
silently swallow some IOError instances. This is arguably the
correct thing to do. However, it had the unfortunate side-effect
of causing StdioError to bubble up to sensitive code like
transaction aborts, leading to an uncaught exceptions and failures
to e.g. roll back a transaction. This could occur when a remote
HTTP or SSH client connection dropped. The new behavior is
resulting in semi-frequent "abandonded transaction" errors on
multiple high-volume repositories at Mozilla.

This commit effectively reverts 45345e9870c3 and b30126fa95bc to
restore the old behavior.

I agree with the principle that I/O errors shouldn't be ignored.
That makes this change... unfortunate. However, our hands are tied
for what to do on stable. I think the proper solution is for the
ui's behavior to be configurable (possibly via a context manager).
During critical sections like transaction rollback and abort, it
should be possible to suppress errors. But this feature would not
be appropriate on stable.
2017-08-15 13:04:31 -07:00
Nathan Goldbaum
0de8be7e62 log: mention ui.logtemplate in the help text 2017-08-16 10:24:49 -05:00
Mike Hommey
73f4a16a4f branchmap: revert a81106261d38 for Python 2.7 compatibility
Old versions of python 2.7 don't like that the second argument to
struct.unpack_from is a bytearray, so the change removing the util.buffer
around that argument in branchmap broke running on older versions of python
2.7.

Differential Revision: https://phab.mercurial-scm.org/D330
2017-08-10 20:47:19 -07:00
Yuya Nishihara
509744ddfc ssh: unban the use of pipe character in user@host:port string
This vulnerability was fixed by the previous patch and there were more ways
to exploit than using '|shellcmd'. So it doesn't make sense to reject only
pipe character.

Test cases are updated to actually try to exploit the bug. As the SSH bridge
of git/svn subrepos are not managed by our code, the tests for non-hg subrepos
are just removed.

This may be folded into the original patches.
2017-08-07 22:22:28 +09:00
Jun Wu
a0e5a4defb ssh: quote parameters using shellquote (SEC)
This patch uses shellquote to quote ssh parameters more strictly to avoid
shell injection.
2017-08-04 23:54:12 -07:00
Sean Farley
39898f2a8a subrepo: add tests for git rogue ssh urls (SEC)
'ssh://' has an exploit that will pass the url blindly to the ssh
command, allowing a malicious person to have a subrepo with
'-oProxyCommand' which could run arbitrary code on a user's machine. In
addition, at least on Windows, a pipe '|' is able to execute arbitrary
commands.

When this happens, let's throw a big abort into the user's face so that
they can inspect what's going on.
2017-07-31 14:55:11 -07:00
Sean Farley
da301ac6a0 subrepo: add tests for svn rogue ssh urls (SEC)
'ssh://' has an exploit that will pass the url blindly to the ssh
command, allowing a malicious person to have a subrepo with
'-oProxyCommand' which could run arbitrary code on a user's machine. In
addition, at least on Windows, a pipe '|' is able to execute arbitrary
commands.

When this happens, let's throw a big abort into the user's face so that
they can inspect what's going on.
2017-07-31 16:44:17 -07:00
Sean Farley
e199b92002 sshpeer: check for safe ssh url (SEC)
Checking in the sshpeer for a rogue ssh:// urls seems like the right
place to do it (instead of whack-a-mole with pull, clone, push, etc).
2017-08-01 14:40:19 -07:00
Augie Fackler
5f2d0af8a2 ssh: ban any username@host or host that starts with - (SEC)
This paranoia probably isn't required, but it can't hurt either.
2017-08-04 14:00:03 -04:00