# skip-blame because this was mechanically rewritten the following script. I
ran it on both *.t and *.py, but none of the *.py changes were proper. All *.t
ones appear to be, and they run without addition failures on both Windows and
Linux.
import argparse
import os
import re
ap = argparse.ArgumentParser()
ap.add_argument('path', nargs='+')
opts = ap.parse_args()
globre = re.compile(r'^(.*) \(glob\)(.*)$')
for p in opts.path:
tmp = p + '.tmp'
with open(p, 'rb') as src, open(tmp, 'wb') as dst:
for line in src:
m = globre.match(line)
if not m or '$LOCALIP' in line or '*' in line:
dst.write(line)
continue
if '?' in line[:-3] or ('?' in line[:-3] and line[-3:] != '(?)'):
dst.write(line)
continue
dst.write(m.group(1) + m.group(2) + '\n')
os.unlink(p)
os.rename(tmp, p)
Before, early options were stripped from args, and because of this, some
kind of parsing errors weren't reported. For example,
$ hg ci -m -Ra file
would execute "hg ci -m file" in repository "a".
This patch fixes the issue by parsing early options again by real getopt-based
parser, and verifying the results. If the early parsing appears wrong, hg just
aborts. The current error message seems not nice, and should be improved, maybe
in V2 or follow-up.
Note that this isn't a security feature because we can still do anything by
using shell aliases.
Upon pull or unbundle, we display a message with the range of new revisions
fetched. This revision range could readily be used after a pull to look out
what's new with 'hg log'. The algorithm takes care of filtering "obsolete"
revisions that might be present in transaction's "changes" but should not be
displayed to the end user.
Previously, blackbox always appends to blackbox.log and creates the
directory for that file on demand. That could be an issue if:
1. chg starts from `$REPO` directory, so `ui._bbrepo` is set.
2. `rm -rf $REPO`.
3. `chg init $REPO`, blackbox writes something and `init` will fail
because `$REPO` directory is non-empty.
This patch fixes that by verifying whether vfs exists before re-using it.
Differential Revision: https://phab.mercurial-scm.org/D768
I've observed some weirdness around this, and needed to rule some
things out. There aren't any bugs in core around this, but it was nice
to have confirmation.
`lastui` decides where (where is the `.hg`) to use if the current `ui`
object does not have a `_bbrepo` associated. Previously it only gets set in
`ui.log`, which means unless a `ui` with repo associated calls `log` with
tracked event, blackbox does not know where to write its log. This patch
makes `reposetup` set `lastui` so it so we could log some more events (see
test changes).
Differential Revision: https://phab.mercurial-scm.org/D655
The added test will show:
$ $PYTHON showsize.py .hg/blackbox*
.hg/blackbox.log: < 500
.hg/blackbox.log.1: < 500
.hg/blackbox.log.2: < 500
.hg/blackbox.log.3: < 500
.hg/blackbox.log.4: < 500
.hg/blackbox.log.5: >= 500
with previous code.
The issue is caused by blackbox caching file objects *by path*, and the
rotation size check could run on a wrong file object (i.e. it should check
"blackbox.log", but `filehandles["blackbox.log"]` contains a file object
that has been renamed to "blackbox.log.5").
This patch removes the "filehandlers" global cache added by 39bd7b0c79fe to
solve the issue.
I think the original patch was trying to make different ui objects use a same
file object if their blackbox.log path is the same. In theory it could also
be problematic in the rotation case. Anyway, that should become unnecessary
after D650.
Differential Revision: https://phab.mercurial-scm.org/D648
Currently when we have multiple heads on the same branch, update tells us that
there some more heads for the current branch but does not tells us the head to
which the repository has been updated to. It makes more sense showing the
head we updated to and then telling there are some more heads.
Now that we garantee that branchmap cache are updated at the end of the
transaction we can drop that one. This removes a problematic case with nested
transaction where the new cache could be written on disk before the transaction
is finished.
The test change is harmless, since we update the cache at a later point, the
dirstate have been updated in between.
The quoting logic here was actually insufficient, and would have had
bogus b-prefixes on Python 3. shellquote seems more appropriate
anyway. Surprisingly, only two tests have output changes, and both of
them look reasonable to me (both are in blackbox logs).
Spotted by Yuya during review.
This should be extremely useful for helping users debug without having
to see their complete configuration.
Shell aliases do not get their expansion logged, because we don't look
and see if we're in a repo before we dive into the execution of a
shell alias. As a result, the ui object doesn't know where to log.
I used test-dispatch.py to demonstrate what would happen if
a log file changed from being readonly to writable, by
having it replace a directory (proxy for readonly/not-writable)
with a log file in between transactions of a running python
process (proxy for Mercurial).
This commit makes it easier for people to follow what the test
is doing, by creating a real file that people can read.
The blackbox test rewrites a copy of test-dispatch.py on the fly, and adds
a couple of lines with the s/// command. GNU sed supports the use of the
\n escape to represent a newline, but not Solaris sed. Using a literal
newline, prefixed by a backslash, works with both versions of the utility.
Without this, anyone creating a ui object using: uimod.ui()
skips the blackbox.
Also, anyone doing ui.copy() skipped the blackbox.
Unfortunately, the ui object lifestyle is a bit messy,
the first one that's created is never actually initialized
with subclasses, instead pieces of the subclass are adopted
into the primal ui object. In order to handle this, a
_partialinit method will be called to ensure that the
blackboxui is properly initialized.
Without this, the last logged entry didn't have access to
the repository, and thus couldn't report its version
(and especially that an add or similar dirtied it).
A side-effect is that one repo leaks until process exit...
Without this, while you could see the list of commands run,
it wasn't possible to identify what they were doing, because commads
could rely on revsets (including remote input which varies over time).
A concern around the user experience of Mercurial is user getting stuck on there
own topological branch forever. For example, someone pulling another topological
branch, missing that message in pull asking them to merge and getting stuck on
there own local branch.
The current way to "address" this concern was for bare 'hg update' to target the
tipmost (also latest pulled) changesets and complain when the update was not
linear. That way, failure to merge newly pulled changesets would result in some
kind of failure.
Yet the failure was quite obscure, not working in all cases (eg: commit right
after pull) and the behavior was very impractical in the common case
(eg: issue4673).
To be able to change that behavior, we need to provide other ways to alert a
user stucks on one of many topological head. We do so with an extra message after
bare update:
1 other heads for branch "default"
Bookmark get its own special version:
1 other divergent bookmarks for "foobar"
There is significant room to improve the message itself, and we should augment
it with hint about how to see theses other heads or handle the situation (see
in-line comment). But having "a" message is already a significant improvement
compared to the existing situation. Once we have it we can iterate on a better
version of it. As having such message is an important step toward changing the
default destination for update and other nicety, I would like to move forward
quickly on getting such message.
This was discussed during London - October 2015 Sprint.
With util.getpid, it is now possible to define fixed pids.
Future iterations can define a map of pids on a locked
first come first serve basis to create a more realistic
harness, but for now this is good enough.
This applies to blackbox, but could apply to other
tests as well.
Without this, when there are multiple ui views, each blackbox
will have its own file handle, and the logging will be in
a really bad order.
Also, because of the way blackbox works, it never closes its
file handles, which means the last output before exit is
often lost.
While it is not easy to make a file 000 on Windows, you can
emulate most of the behaviors by replacing the file with a directory.
Also corrects test description to properly indicate that failing to
read from the log is fatal.
c67339617276 (while 3.4 code-freeze) made all 'update' hooks run after
releasing wlock for visibility of in-memory dirstate changes. But this
breaks paired invocation of 'preupdate' and 'update' hooks.
For example, 'hg backout --merge' for TARGET revision, which isn't
parent of CURRENT, consists of steps below:
1. update from CURRENT to TARGET
2. commit BACKOUT revision, which backs TARGET out
3. update from BACKOUT to CURRENT
4. merge TARGET into CURRENT
Then, we expects hooks to run in the order below:
- 'preupdate' on CURRENT for (1)
- 'update' on TARGET for (1)
- 'preupdate' on BACKOUT for (3)
- 'update' on CURRENT for (3)
- 'preupdate' on TARGET for (4)
- 'update' on CURRENT/TARGET for (4)
But hooks actually run in the order below:
- 'preupdate' on CURRENT for (1)
- 'preupdate' on BACKOUT for (3)
- 'preupdate' on TARGET for (4)
- 'update' on TARGET for (1), but actually on CURRENT/TARGET
- 'update' on CURRENT for (3), but actually on CURRENT/TARGET
- 'update' on CURRENT for (4), but actually on CURRENT/TARGET
Root cause of the issue focused by c67339617276 is that external
'update' hook process can't view in-memory changes (especially, of
dirstate), because they aren't written out until the end of
transaction (or wlock).
Now, hooks can be invoked just after updating, because previous
patches made in-memory changes visible to external process.
This patch may break backward compatibility from the point of view of
"scheduling hook execution", but should be reasonable because 'update'
hooks had been executed in this order before 3.4.
This patch tests "hg backout" and "hg unshelve", because the former
activates the transaction before 'update' hook invocation, but the
former doesn't.
This adds the process id to the line header for the blackbox output. This is
useful for distinguishing processes when using the blackbox on a server and many
processes are writing to the blackbox at once.
The update command is touching the repository and should lock it for
the length of its operations. Equally importantly, it should lock the
repository when it is writing bookmarks. It wasn't doing so until now,
leaving doors open for all kinds of drunk beaver parties.
This results in some minor tests changes, and the fixing of a couple
of bugs from race conditions.
Code does not receive any changes beside extra indentation.
We now have multiple tags cache files. Record exactly which file is
being written. This should help aid debugging into performance issues
with any single filter.
.hgtags fnodes are now written to a shared cache file. They don't need
to exist in the per-filter tags cache files. Stop writing them.
The format of the tags cache file has changed in a backwards
incompatible way. This should be acceptable, as we just established
per-filter tags cache files and no client should have per-filter tags
cache files that will need to be read. So no backwards compatbility
concern is present.
The new format has a single header line followed by resolved tags
entries.
The header line is similar to the old first line with a major
difference: we now compute and store a hash of the filtered revisions.
Before, if the set of filtered revs changed, we may return incorrect
results. We now detect that.
A test for verifying filtered rev change is handled properly has been
added.
Having all blackbox log testing in test-blackbox.t isn't scalable. Move
the mock blackbox extension into its own file so we can start to move
blackbox logging into other tests.
This adds a new root hghave to test against. Almost all of these are a
subset of unix-permissions, but that is also used for checking exec
bit handling.
Previously the blackbox wrapped runcommand, but this failed to see the error
codes that were created if an exception occurred. I moved that logging to now
wrap _runcatch, so it can observe and log the actual error code (such as when
a user ctrl+c's during a command).
Updated the tests as well. Tested the change by running all the tests with the
blackbox extension enabled and verifying nothing broke (aside from things that
printed what extensions were enabeld).
The progress tests are affected by calls to time.time() so they needed to be
updated to pass.
The windows and vfat test runs were failing due to read/write permissions not
working the same on those systems. On vfat, permissions can't be changed
at all, and on windows it seems the chmod emulation doesn't remove read
permissions. We could theoretically get the 'cannot write to blacklog.log'
test to pass on windows but there's no #if condition to let us exclude vfat
only.
Verified that test-blackbox passes on windows now.
If enabled, log rotation prevents the amount of space used by the
blackbox log from growing without bound. This becomes important in
cases where there are a lot of busy repositories managed by humans
and automation on many machines.
In large deployments, we cannot reasonably track all the repos where
blackbox logs need to be managed, so it is safer to have blackbox
manage its own logs than to move responsibility to an external tool
such as logrotate.
This change adds two configuration keys:
* blackbox.maxsize is the maximum allowable size of the current log
* blackbox.maxfiles is the number of log files to maintain
Writes the backup bundle paths to the blackbox so it's easy to see which
backup bundle is associated with which command when you are debugging an
issue.
Example output:
2013/03/13 10:39:56 durham> strip tip
2013/03/13 10:39:59 durham> saved backup bundle to /data/users/durham/www-hg/.hg/strip-backup/e5fac262363a-backup.hg
2013/03/13 10:40:03 durham> strip tip exited 0 after 7.97 seconds
The blackbox was logging every head after every incoming group.
Now we only log the heads that have changed.
Added a test. Moved the hooks test to the bottom of the file since
the hooks interfer with the tests after it.
