> For some reason, I thought someone (Mads?) said we had basically given
> up on ui.slash, and some commands didn't support it.
(from https://bz.mercurial-scm.org/show_bug.cgi?id=5572#c1)
So the ui.slash option doesn't always work and is somewhat confusing. Let's
make it clearer we won't improve the situation.
This will allow developers to test against various server implementations. I
didn't put it under [devel] because it's possible that some user needs to use it
in the field.
As we were trying to transition off of the non production lfs-test-server for
further experimenting, one of the problems we ran into was interoperability. A
coworker setup gitbucket[1] to act as the blob server, tested with git, and
passed it off to me. But push failed with a message saying "abort: LFS server
returns invalid JSON:", and then proceeded to dump a huge HTML page to the
screen. It turns out that it is assuming that git is the only thing that wants
to do a blob transfer, and everything else is a web browser wanting HTML.
It's only a single data point, but I suspect other things may be doing this too.
RFC7231 gives an example [2] of listing multiple products in decreasing order of
significance. Since the standard provides for this, and since it works with the
one problematic server I found, I'm just enabling this by default for a better
UX.
There's nothing significant about the version of git chosen, other than it is
the current version.
[1] https://github.com/gitbucket/gitbucket/
[2] https://tools.ietf.org/html/rfc7231#page-46
Make 'hg outgoing' respect "paths.default:pushurl" in addition to
"paths.default-push".
'hg outgoing' has always meant "what will happen if I run 'hg push'?" and it's
still documented that way:
Show changesets not found in the specified destination repository or the
default push location. These are the changesets that would be pushed if a
push was requested.
If the user uses the now-deprecated "paths.default-push" path, it continues to
work that way. However, as described at
https://bz.mercurial-scm.org/show_bug.cgi?id=5365, it doesn't behave the same
with "paths.default:pushurl".
Why does it matter? Similar to the bugzilla reporter, I have a read-only mirror
of a non-Mercurial repository:
upstream -> imported mirror -> user clone
^-----------------------/
Users push directly to upstream, and that content is then imported into the
mirror. However, those repositories are not the same; it's possible that the
mirroring has either broken completely, or an import process is running and not
yet complete. In those cases, 'hg outgoing' will list changesets that have
already been pushed.
Mozilla's desired behavior described in bug 5365 can be accomplished through
other means (e.g. 'hg outgoing default'), preserving the consistency and
meaning of 'hg outgoing'.
Once we're ready to turn this functionality on more widely, we might
want to write a symlink-free version of the test that can run on
Windows, but for now we'll just leave it disabled there.
Differential Revision: https://phab.mercurial-scm.org/D1710
We already have the ability to customize the ssh command line arguments, let's
add the ability to customize its environment as well.
Example use-case is ssh.exe from Git on Windows. If `HOME` enviroment variable
is present and has some non-empty value, ssh.exe will try to access that
location for some stuff (for example, it seems for resolving `~` in
`.ssh/config`). Git for Windows seems to sometimess set this variable to the
value of `/home/username` which probably works under Git Bash, but does not
work in a native `cmd.exe` or `powershell`. Whatever the root cause, setting
`HOME` to be an empty string heals things. Therefore, some distributors
might want to set `sshenv.HOME=` in the configuration (seems less intrusive
that forcing everyone to tweak their env).
Test Plan:
- rt
Differential Revision: https://phab.mercurial-scm.org/D1683
I don't see why unamend should be disallowed when allowunstable is
set. By switching to rewriteutil.precheck() we fix that and get more
consistent error messages (and some additional ones).
Differential Revision: https://phab.mercurial-scm.org/D1682
This significantly speeds up lfs prefetch. With fast network we are
seeing ~50% improvement of overall prefetch times
Because of worker's API in posix we do lose finegrained progress update and only
see progress when a file finished downloading.
Test Plan:
Run tests:
./run-tests.py -l test-lfs*
....
# Ran 4 tests, 0 skipped, 0 failed.
Run commands resulting in lfs prefetch e.g. hg sparse --enable-profile
Differential Revision: https://phab.mercurial-scm.org/D1568
The windows workers weren't daemons and were not correctly killed when ctrl-c'd from the terminal. Withi this change when the main thread is killed, all daemons get killed as well.
I also reduced the time we give to workers to cleanup nicely to not have people ctrl-c'ing when they get inpatient.
The output when threads clened up nicely:
PS C:\<dir>> hg.exe sparse --disable-profile SparseProfiles/<profile>.sparse
interrupted!
The output when threads don't clenup in 1 sec:
PS C:\<dir> hg.exe sparse --enable-profile SparseProfiles/<profile>.sparse
failed to kill worker threads while handling an exception
interrupted!
Exception in thread Thread-4 (most likely raised during interpreter shutdown):
PS C:\<dir>>
Test Plan:
Run hg command on windows (pull/update/sparse). Ctrl-C'd sparse --enable-profile command that was using threads and observed in proces explorer that all threads got killed.
ran tests on CentOS
Differential Revision: https://phab.mercurial-scm.org/D1564
This adds config to disable/enable workers with default being enabled.
Test Plan:
enabled profile without updaing .hg/hgrc (the default should be to use workers) and ran
hg sprase --enable-profile <profile>.sparse
Watched in the proces explorer that hg started 12 new threads for materializing files (this is my worker.numcpus) value
Added
[worker]
enabled = False
to the .hg/hgrc and re ran the command. This time hg didn't spawn any new threads for matreializing of files
Differential Revision: https://phab.mercurial-scm.org/D1460
This adds handling of exceptions from worker threads and resurfaces them as if the function ran without workers.
If any of the threads throws, the main thread kills all running threads giving them 5 sec to handle the interruption and raises the first exception received.
We don't have to join threads if is_alive() is false
Test Plan:
Ran multiple updates/enable/disable sparse profile and things worked well
Ran test on CentOS- all tests passing on @ passed here
Added a forged exception into the worker code and got it properly resurfaced and the rest of workers killed: P58642088
PS C:\open\<repo>> ..\facebook-hg-rpms\build\hg\hg.exe --config extensions.fsmonitor=! sparse --enable-profile <profile>
updating [==> ] 1300/39166 1m57sException in thread Thread-3:
Traceback (most recent call last):
File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\threading.py", line 801, in __bootstrap_inner
self.run()
File "C:\open\facebook-hg-rpms\build\hg\mercurial\worker.py", line 244, in run
raise e
Exception: Forged exception
Exception in thread Thread-2:
Traceback (most recent call last):
File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\threading.py", line 801, in __bootstrap_inner
self.run()
File "C:\open\facebook-hg-rpms\build\hg\mercurial\worker.py", line 244, in run
raise e
Exception: Forged exception
<...>
Traceback (most recent call last):
File "C:\open\facebook-hg-rpms\build\hg\hgexe.py", line 41, in <module>
dispatch.run()
File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 85, in run
status = (dispatch(req) or 0) & 255
File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 173, in dispatch
ret = _runcatch(req)
File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 324, in _runcatch
return _callcatch(ui, _runcatchfunc)
File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 332, in _callcatch
return scmutil.callcatch(ui, func)
File "C:\open\facebook-hg-rpms\build\hg\mercurial\scmutil.py", line 154, in callcatch
return func()
File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 314, in _runcatchfunc
return _dispatch(req)
File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 951, in _dispatch
cmdpats, cmdoptions)
File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\remotefilelog\__init__.py", line 415, in runcommand
return orig(lui, repo, *args, **kwargs)
File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\undo.py", line 118, in _runcommandwrapper
result = orig(lui, repo, cmd, fullargs, *args)
File "C:\open\facebook-hg-rpms\build\hg\hgext\journal.py", line 84, in runcommand
return orig(lui, repo, cmd, fullargs, *args)
File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\perftweaks.py", line 268, in _tracksparseprofiles
res = runcommand(lui, repo, *args)
File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\perftweaks.py", line 256, in _trackdirstatesizes
res = runcommand(lui, repo, *args)
File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\copytrace.py", line 144, in _runcommand
return orig(lui, repo, cmd, fullargs, ui, *args, **kwargs)
File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\fbamend\hiddenoverride.py", line 119, in runcommand
result = orig(lui, repo, cmd, fullargs, *args)
File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 712, in runcommand
ret = _runcommand(ui, options, cmd, d)
File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 959, in _runcommand
return cmdfunc()
File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 948, in <lambda>
d = lambda: util.checksignature(func)(ui, *args, **strcmdopt)
File "C:\open\facebook-hg-rpms\build\hg\mercurial\util.py", line 1183, in check
return func(*args, **kwargs)
File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\fbsparse.py", line 860, in sparse
disableprofile=disableprofile, force=force)
File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\fbsparse.py", line 949, in _config
len, _refresh(ui, repo, oldstatus, oldsparsematch, force))
File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\fbsparse.py", line 1116, in _refresh
mergemod.applyupdates(repo, typeactions, repo[None], repo['.'], False)
File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\remotefilelog\__init__.py", line 311, in applyupdates
return orig(repo, actions, wctx, mctx, overwrite, labels=labels)
File "C:\open\facebook-hg-rpms\build\hg\mercurial\merge.py", line 1464, in applyupdates
for i, item in prog:
File "C:\open\facebook-hg-rpms\build\hg\mercurial\worker.py", line 286, in _windowsworker
raise t.exception
Exception: Forged exception
PS C:\open\ovrsource>
Differential Revision: https://phab.mercurial-scm.org/D1459
This change implements thread based worker on windows.
The handling of exception from within threads will happen in separate diff.
The worker is for now used in mercurial/merge.py and in lfs extension
After multiple tests and milions of files materiealized, thousands lfs fetched
it seems that neither merge.py nor lfs/blobstore.py is thread unsafe. I also
looked through the code and besides the backgroundfilecloser (handled in base
of this) things look good.
The performance boost of this on windows is
~50% for sparse --enable-profile
* Speedup of hg up/rebase - not exactly measured
Test Plan:
Ran 10s of hg sparse --enable-profile and --disable-profile operations on large profiles and verified that workers are running. Used sysinternals suite to see that all threads are spawned and run as they should
Run various other operations on the repo including update and rebase
Ran tests on CentOS and all tests that pass on @ pass here
Differential Revision: https://phab.mercurial-scm.org/D1458
This disables background file closing when in not in main thread
Test Plan:
Ran pull, update, sparse commands and watched the closer threads created and destroyed in procexp.exe
ran test on CentOS. No tests broken compared to the base
Differential Revision: https://phab.mercurial-scm.org/D1457
This does a few things:
* Changes "-r" to "--rev", since "-r" is not a valid short form
* Removes non-existent "-l" and "-b" options
* Removes "..." after options, since we don't usually have that
Differential Revision: https://phab.mercurial-scm.org/D1706
I may add an alternative way of getting copy metadata (from changelog,
not filelog) but the chaining with the dirstate copy metadata will be
the same, so it will probably help to have this extracted. Even if
that doesn't happen, the next patch will show that we can simplify
this a bit after this refactoring, so it seems worth it regardless.
Differential Revision: https://phab.mercurial-scm.org/D1697
The function would ignore the matcher if the dirstate copies were
requested. It doesn't matter in practice because all callers used the
returned map only for looking up specific files from and those files
had already been filtered by the matcher (AFACT). Still, it's a little
confusing, so let's make it clearer by respecting the matcher in this
case too.
Differential Revision: https://phab.mercurial-scm.org/D1695
It seems like it didn't even exist when debugdiscovery was introduced
in 43f4c1113c8d (discovery: add new set-based discovery, 2011-05-02).
Differential Revision: https://phab.mercurial-scm.org/D1693
It seems like it didn't even exist when debugdiscovery was introduced
in 43f4c1113c8d (discovery: add new set-based discovery, 2011-05-02).
Differential Revision: https://phab.mercurial-scm.org/D1692
It seems like it didn't even exist when debugdiscovery was introduced
in 43f4c1113c8d (discovery: add new set-based discovery, 2011-05-02).
Differential Revision: https://phab.mercurial-scm.org/D1691
Some shells' echo (e.g. Bash's) do not interpret backslash escapes by
default, while others shells' do (e.g. Dash's). This led to a failure
in test-run-tests.t. We could switch to the more compatible printf,
but using a heredoc seems simpler, so that's what this patch does.
For anyone who wants to test this for themselves, remove the "unset
HGTEST_SHELL" line in helper-runtests.sh and pass
e.g. "--shell=/bin/dash" to the test runner.
Differential Revision: https://phab.mercurial-scm.org/D1699
Once upon a time, in 1995, there were browsers that didn't understand <script>
tags and they would simply show the code inside as text. This started a
tradition of wrapping everything inside <script> in <!-- HTML comments -->.
Nowadays, it's not only not needed, but can be considered harmful[1]:
- within XHTML documents, the source will actually be hidden from all browsers
and rendered useless
- `--` is not allowed within HTML comments, so any decrement operations in
script are invalid
[1]: http://www.javascripttoolbox.com/bestpractices/#comments
The branch information was properly preserved in the changeset, but the
"active" branch of the working copy could be lost (the branch of the base
being used).
Histedit used to behave properly in this regard but the case was not tested
and regressed 4 years ago in c038baa4b6f0.
This effectively conditionalizes 7ac081713920. Some Linux distributions (like
CentOS 7) use really old versions, and the change referenced was causing
exceptions to be thrown.
Even though the deprecation warning says 'since 2.5.0', it wasn't marked as such
in 2.5.1, but is by 2.6.0. This was tested with 2.4.2 and 2.6.0 with
PYTHONWARNINGS=::DeprecationWarning, and both paths were exercized.
In large repositories, updates involving the creation of many files check the
same directories repeatedly in the wctx manifest. Move these checks out to a
separate loop to avoid repeated checks hitting the manifest.
Differential Revision: https://phab.mercurial-scm.org/D1226
As mentioned in D1222, the recent pathconflicts change regresses update
performance in large repositories when many files are being updated.
To mitigate this, we introduce two caches of directories that have
already found to be either:
- unknown directories, but which are not aliased by files and
so don't need to be checked if they are files again; and
- missing directores, which cannot cause path conflicts, and
cannot contain a file that causes a path conflict.
When checking the paths of a file, testing against this caches means we can
skip tests that involve touching the filesystem.
Differential Revision: https://phab.mercurial-scm.org/D1224
If this feature is enabled, early options are parsed using the global options
table. As the parser stops processing options when non/unknown option is
encountered, it won't mistakenly take an option value as a new early option.
Still "--" can be injected to terminate the parsing (e.g. "hg -R -- log"), I
think it's unlikely to lead to an RCE.
To minimize a risk of this change, new fancyopts.earlygetopt() path is enabled
only when +strictflags is set. Also the strict parser doesn't support '--repo',
a short for '--repository' yet. This limitation will be removed later.
As this feature is backward incompatible, I decided to add a new opt-in
mechanism to HGPLAIN. I'm not pretty sure if this is the right choice, but
I'm thinking of adding +feature/-feature syntax to HGPLAIN. Alternatively,
we could add a new environment variable. Any bikeshedding is welcome.
Note that HGPLAIN=+strictflags doesn't work correctly in chg session since
command arguments are pre-processed in C. This wouldn't be easily fixed.
The next patch will add a flag for strict parsing of early options, where
we'll have to parse all early options at once instead of processing them
one-by-one by dispatch._earlygetopt(). That's why I decided to hook
fancyopts().
All dispatch._early*opt() functions is planned to be replaced with this
function. But in this stable series, only the strict mode will be handled
by fancyopts.earlygetopt().
Previously, the largefile for a dropped standin would be deleted here, and then
restored from the cache. This had the effect of clobbering uncommitted changes
if a revert caused the file to be forgotten, which is not what happens with a
normal file. Now the removal and update is skipped for dropped largefiles, and
the corresponding standin is deleted from disk.
This was noticed when working on issue5738 because the forgotten standin files
were left behind, and that changes the behavior of the next rename to that
directory. My first attempt was to cleanup the standins before calling this.
That failed, because this function deletes the largefile if the corresponding
standin is missing.
This function is called by the revert command, merge (and therefore update), and
patch, via the scmutil.marktouched() override. So it should be pretty narrow in
scope.
I didn't mark issue5738 as fixed because the move related issues can still
happen if the main tree and the .hglf subtree get out of sync somehow. I don't
see an easy fix for that, but that should be an edge case. If whoever queues
this thinks it is good enough to close out the bug and can cram it into the
summary, go for it.
These things were uncovered looking at issue5738.
First, if the destination directory exists under .hglf, the source is moved
under the destination instead of renaming the last component for `hg mv srcdir
dstdir`. This is extra confusing, because it occurs even if the user visible
destination (i.e. the path _not_ under .hglf) does not exist.
Additionally, when a largefile is forgotten via revert, any modifications end up
getting clobbered. For normal files, the forgotten file is left unchanged, as
shown by test-import.t. The forget command on a largefile will correctly leave
the file unmodified.
The `hg lfconvert --to-normal` command uses the convert extension internally to
work its magic, but that produced devel-warn messages if the convert extension
wasn't loaded by the user. The test in 658e7a6d93e0 (modified here) wasn't
showing the warnings because the convert extension was loaded via $HGRCPATH.
Most of the config options default to None/False, but 'hg.usebranchnames' and
'hg.tagsbranch' are supposed to default to True and 'default' respectively.
The first iteration of this was to ui.setconfig() inside lfconvert, to force the
convert extension to load. But there really is no precedent for doing this, and
check-config complained that 'extensions.convert' isn't documented. Yuya
suggested this alternative.
This partially backs out 448e09d8859d.
Repoview can have a different life cycle, causing issue in some corner
cases. The particular instance that revealed this comes from localpeer. The
localpeer hold a reference to the unfiltered repository, but calling 'local()'
will create an on-demand 'visible' repoview. That repoview can be garbaged
collected any time. Here is a simplified step by step reproduction::
1) tr = peer.local().transaction('foo')
2) tr.close()
After (1), the repoview object is garbage collected, so weakref used in (2)
point to nothing.
Thanks to Sean Farley for helping raising and debugging this issue.
Before, early options were stripped from args, and because of this, some
kind of parsing errors weren't reported. For example,
$ hg ci -m -Ra file
would execute "hg ci -m file" in repository "a".
This patch fixes the issue by parsing early options again by real getopt-based
parser, and verifying the results. If the early parsing appears wrong, hg just
aborts. The current error message seems not nice, and should be improved, maybe
in V2 or follow-up.
Note that this isn't a security feature because we can still do anything by
using shell aliases.
So we can easily compare it with the corresponding getopt() result.
There's a minor behavior change. Before, "hg --cwd ''" failed with ENOENT.
But with this patch, an empty cwd is silently ignored. "hg -R ''" has always
worked as such, so -R has no BC.
This allows us to parse the original args later by full-blown getopt() in
order to verify the result of the faulty early parsing. Still we need the
'strip=True' behavior for shell aliases.
Note that this series is RFC because it seems to change too much to be
included in stable release.
