This used to be needed to paper over hashlib not being in all Pythons
we support, but that's not a problem anymore, so we can simplify
things a little bit.
The cPickle is renamed to _pickle in python3 and this C extension is available
in pickle which was not included in earlier versions. So imports are conditionalized
to import cPickle in py2 and pickle in py3. Moreover the use of pickle in py2 is
switched to cPickle as the C extension is faster. The hack is added in util.py and
the modules import util.pickle
Upcoming patches will teach host fingerprint checking to verify
non-SHA1 fingerprints.
Many x509 certificates these days are SHA-256. And modern browsers
often display the SHA-256 fingerprint for certificates. Since
SHA-256 fingerprints are highly visible and easy to obtain, we
want to support them for fingerprint pinning. So add SHA-256
support to util.
I did not add SHA-256 to DIGESTS and DIGESTS_BY_STRENGTH because
this will advertise the algorithm on the wire protocol. I wasn't
sure if that would be appropriate. I'm playing it safe by leaving
it out for now.
In some cases below, copying from backup is used to restore original
contents of a file. If copying keeps ctime, mtime and size of a file,
restoring is overlooked, and old contents cached before restoring
isn't invalidated as expected.
- failure of transaction before closing (from '.hg/journal.backup.*')
- rollback of previous transaction (from '.hg/undo.backup.*')
To avoid such problem, this patch makes copyfile() avoid ambiguity of
file stat, if needed.
Ambiguity check is executed, only if:
- checkambig=True is specified (not all copying needs ambiguity check), and
- destination file exists before copying
This patch also adds 'not (copystat and checkambig)' assertion,
because combination of copystat and checkambig is meaningless.
This patch is a part of preparation for "Exact Cache Validation Plan":
https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan
Ambiguity check is executed at close(), only if:
- atomictempfile is created with checkambig=True, and
- target file exists before renaming
This restriction avoids performance decrement by needless examination
of file stat (for example, filelog doesn't need exact cache
validation, even though it uses atomictempfile to write changes out).
See description of filestat class for detail about why the logic in
this patch works as expected.
This patch is a part of preparation for "Exact Cache Validation Plan":
https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan
Current posix.cachestat implementation might overlook change of a
file, if changing keeps ctime, mtime and size of file. Comparison of
inode number also overlooks changing in such situation, because inode
number is rapidly reused.
Contents of a file cached before changing isn't invalidated as
expected, if change of a file is overlooked for this "ambiguity" of
file stat.
This patch adds filestat class to detect ambiguity of file stat.
This patch is a part of preparation for "Exact Cache Validation Plan":
https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan
Update makedirs() to ignore EEXIST in case someone else has already created the
directory in question. Previously the ensuredirs() function existed, and was
nearly identical to makedirs() except that it fixed this race. Unfortunately
ensuredirs() was only used in 3 places, and most code uses the racy makedirs()
function. This fixes makedirs() to be non-racy, and replaces calls to
ensuredirs() with makedirs().
In particular, mercurial.scmutil.origpath() used the racy makedirs() code,
which could cause failures during "hg update" as it tried to create backup
directories.
This does slightly change the behavior of call sites using ensuredirs():
previously ensuredirs() would throw EEXIST if the path existed but was a
regular file instead of a directory. It did this by explicitly checking
os.path.isdir() after getting EEXIST. The makedirs() code did not do this and
swallowed all EEXIST errors. I kept the makedirs() behavior, since it seemed
preferable to avoid the extra stat call in the common case where this directory
already exists. If the path does happen to be a file, the caller will almost
certainly fail with an ENOTDIR error shortly afterwards anyway. I checked
the 3 existing call sites of ensuredirs(), and this seems to be the case for
them.
python3 url.request and url.error are mapped as util.urlreq/util.urlerr
python2 equivalents from urllib/urllib2 are mapped according to the py3
hierarchy
DVCS are very useful to store various texts (as legislation) written before
Unix epoch. Fri, 13 Dec 1901 is a nice gain over Thu, 01 Jan 1970.
Revert 856a0e92d107 and b47a679b4e83, fix b2228cbaa635. Add tests.
While the pycompat module will actually handle divergence, please
access these properties from the util module:
util.queue = Queue.Queue / queue.Queue
util.empty = Queue.Empty / queue.Empty
Previously, 'hg serve -d' was trying to exec the bundled python executable,
which failed with:
Unknown option: --
usage: python [option] ...
Try 'python -h'...
abort: child process failed to start
See the previous patch for details about the content of the various command
variables. Note that unlike the previous patch here an application bundling
Mercurial could set $HG in the environment to get the correct result, there
isn't anything that a bundling application could do to get the correct result
here.
'hg serve -d' now launches under TortoiseHg, and there is a process listed in
the background, but a client process cannot connect to it for some reason, so
more investigation is needed.
sys.executable is "$appbundle/Contents/MacOS/python" when Mercurial is bundled
in a frozen app bundle on OS X, so that isn't appropriate. It appears that this
was only visible for things launched via util.system(), like external hooks,
where $HG was set wrong.
It appears that Mercurial also uses 'sys.modules['__main__'].__file__' (here)
and 'sys.argv[0]' (in platform.gethgcmd()) to figure out the command to spawn.
In both cases, this points to "$appbundle/Contents/Resources/hg", which invokes
the system python since "/usr/bin/env python" is on the shebang line. On my
system with a screwed up python install, I get an error importing the os module
if this script is invoked.
We could take the dirname of sys.executable and join 'hg' instead of this if we
want to be paranoid, but py2app boostrap is setting the environment variable
since 0.1.6 (current version is 0.9), so it seems safe and we might as well use
it.
Apparently unlike py2exe, py2app copies the Mercurial source tree as-is to a
Contents/Resources subdirectory of an app bundle, and places its binary stub in
Contents/MacOS. (The Windows install has the 'hgext' and 'mercurial' modules in
'lib/library.zip', while the help and templates subdirectories have been moved
out of the mercurial directory to the root of the installation. I assume that
the python code living in a zip file is why "py2exe doesn't support __file__".)
Therefore, prior to this change, Mercurial in a frozen app bundle on OS X would
go looking for help *.txt, templates and locale info in Contents/MacOS, where
they don't exist.
There are only a handful of places that test for frozen, and not all of them are
wrong for OS X, so it seems wiser to handle them on a case by case basis, rather
that try to change mainfrozen(). The remaining cases are:
1) util.hgexecutable() wrongly points to the bundled python executable, and
affects $HG in util.system() launched processes (e.g. external hooks)
2) util.hgcmd() wrongly points to the bundled python executable, but it seems
to only affect 'hg serve -d'
3) hook._pythonhook() may be OK, since I didn't see anything outrageous when
printing sys.path from an internal hook. I'm not sure if this special
case is needed on OS X though.
4) sslutil._plainapplepython() is OK, because sys.executable is not
/usr/bin/python, nor is it in /System/Library/Frameworks
This is similar in spirit to contextlib.nested in Python <= 2.6,
but uses an extra level of indirection to avoid its inability to
clean up if an __enter__ method raises an exception.
Why add this mechanism? It greatly simplifies scoped resource
management, and lets us eliminate several hundred lines of try/finally
blocks. In many of these cases the "finally" is separated from the
"try" by hundreds of lines of code, which makes the connection
between resource acquisition and disposal difficult to follow.
(The preferred mechanism would be the "multi-with" syntax of 2.7+,
but Mercurial can't move to 2.7 for a while.)
Intended use:
>>> with ctxmanager(lambda: file('foo'), lambda: file('bar')) as c:
>>> f1, f2 = c()
This will open both foo and bar when c() is invoked, and will close
both upon exit from the block. If the attempt to open bar raises
an exception, the block will not be entered - but foo will still
be closed.
An old implementation of this class (possibly only in my local repo)
allocated nodes in the cache during construction time, making
__init__ slow for large cache capacities. The current implementation
lazily grow the cache size, making this comment wrong.
This diff implements the standard dict copy() method for lrucachedicts, which
will be used in the pushrebase extension to make a copy of the manifestcache.
Contrary to the comment, I didn't see any evidence that we were copying
atime/mtime at all. This adds a parameter to copyfile to optionally copy it and
other stat data, with the default being to not copy it.
Many systems don't support changing the timestamp of a symlink, but we don't
need that in general anyway -- copystat is mostly useful for editors, most of
which will dereference symlinks anyway.
As part of attempting to more aggressively use the existing
lrucachedict, collections.deque operations were frequently
showing up in profiling output, negating benefits of caching.
Searching the internet seems to tell me that the most efficient
way to implement an LRU cache in Python is to have a dict indexing
the cached entries and then to use a doubly linked list to track
freshness of each entry. So, this patch replaces our existing
lrucachedict with a version using such a pattern.
The recently introduced perflrucachedict command reveals the
following timings for 10,000 operations for the following cache
sizes for the existing cache:
n=4 init=0.004079 gets=0.003632 sets=0.005188 mixed=0.005402
n=8 init=0.004045 gets=0.003998 sets=0.005064 mixed=0.005328
n=16 init=0.004011 gets=0.004496 sets=0.005021 mixed=0.005555
n=32 init=0.004064 gets=0.005611 sets=0.005188 mixed=0.006189
n=64 init=0.003975 gets=0.007684 sets=0.005178 mixed=0.007245
n=128 init=0.004121 gets=0.012005 sets=0.005422 mixed=0.009471
n=256 init=0.004143 gets=0.020295 sets=0.005227 mixed=0.013612
n=512 init=0.004039 gets=0.036703 sets=0.005243 mixed=0.020685
n=1024 init=0.004193 gets=0.068142 sets=0.005251 mixed=0.033064
n=2048 init=0.004070 gets=0.133383 sets=0.005160 mixed=0.050359
n=4096 init=0.004053 gets=0.265194 sets=0.004868 mixed=0.048352
n=8192 init=0.004087 gets=0.542218 sets=0.004562 mixed=0.032753
n=16384 init=0.004106 gets=1.064055 sets=0.004179 mixed=0.020367
n=32768 init=0.004034 gets=2.097620 sets=0.004260 mixed=0.013031
n=65536 init=0.004108 gets=4.106390 sets=0.004268 mixed=0.010191
As the data shows, the existing cache's retrieval performance
diminishes linearly with cache size. (Keep in mind the microbenchmark
is testing 100% cache hit rate.)
The new cache implementation reveals the following:
n=4 init=0.006665 gets=0.006541 sets=0.005733 mixed=0.006876
n=8 init=0.006649 gets=0.006374 sets=0.005663 mixed=0.006899
n=16 init=0.006570 gets=0.006504 sets=0.005799 mixed=0.007057
n=32 init=0.006854 gets=0.006459 sets=0.005747 mixed=0.007034
n=64 init=0.006580 gets=0.006495 sets=0.005740 mixed=0.006992
n=128 init=0.006534 gets=0.006739 sets=0.005648 mixed=0.007124
n=256 init=0.006669 gets=0.006773 sets=0.005824 mixed=0.007151
n=512 init=0.006701 gets=0.007061 sets=0.006042 mixed=0.007372
n=1024 init=0.006641 gets=0.007620 sets=0.006387 mixed=0.007464
n=2048 init=0.006517 gets=0.008598 sets=0.006871 mixed=0.008077
n=4096 init=0.006720 gets=0.010933 sets=0.007854 mixed=0.008663
n=8192 init=0.007383 gets=0.015969 sets=0.010288 mixed=0.008896
n=16384 init=0.006660 gets=0.025447 sets=0.011208 mixed=0.008826
n=32768 init=0.006658 gets=0.044390 sets=0.011192 mixed=0.008943
n=65536 init=0.006836 gets=0.082736 sets=0.011151 mixed=0.008826
Let's go through the results.
The new cache takes longer to construct. ~6.6ms vs ~4.1ms. However,
this is measuring 10,000 __init__ calls, so the difference is
~0.2us/instance. We currently only create lrucachedict for manifest
instances, so this regression is not likely relevant.
The new cache is slightly slower for retrievals for cache sizes
< 1024. It's worth noting that the only existing use of lurcachedict
is in manifest.py and the default cache size is 4. This regression
is worrisome. However, for n=4, the delta is ~2.9s for 10,000 lookups,
or ~0.29us/op. Again, this is a marginal regression and likely not
relevant in the real world. Timing `hg log -p -l 100` for
mozilla-central reveals that cache lookup times are dominated by
decompression and fulltext resolution (even with lz4 manifests).
The new cache is significantly faster for retrievals at larger
capacities. Whereas the old implementation has retrieval performance
linear with cache capacity, the new cache is constant time until much
larger values. And, when it does start to increase significantly, it
is a few magnitudes faster than the current cache.
The new cache does appear to be slower for sets when capacity is large.
However, performance is similar for smaller capacities. Of course,
caches should generally be optimized for retrieval performance because
if a cache is getting more sets than gets, it doesn't really make
sense to cache. If this regression is worrisome, again, taking the
largest regression at n=65536 of ~6.9ms for 10,000 results in a
regression of ~0.68us/op. This is not significant in the grand scheme
of things.
Overall, the new cache is performant at retrievals at much larger
capacity values which makes it a generally more useful cache backend.
While there are regressions, their absolute value is extremely small.
Since we aren't using lrucachedict aggressively today, these
regressions should not be relevant. The improved scalability of
lrucachedict should enable us to more aggressively utilize
lrucachedict for more granular caching (read: higher capacity caches)
in the near future. The impetus for this patch is to establish a cache
of decompressed revlog revisions, notably manifest revisions. And since
delta chains can grow to >10,000 and cache hit rate can be high, the
improved retrieval performance of lrucachedict should be relevant.
hashlib was added in Python 2.5. As far as I can tell, SHA-512 is always
available in 2.6+. So move the hashlib import to the top of the file and
remove the one-off handling of SHA-512.
We were computing the quotient and remainder of a division operation
separately. The built-in divmod() function allows us to do this with
a single function call. Do that.
We've globablly forced stat to return integer times which agrees with
our extension code, so this is no longer needed.
This speeds up status on mozilla-central substantially:
$ hg perfstatus
! wall 0.190179 comb 0.180000 user 0.120000 sys 0.060000 (best of 53)
$ hg perfstatus
! wall 0.275729 comb 0.270000 user 0.210000 sys 0.060000 (best of 36)
Alternate fix for this issue which avoids putting extra function calls
and exception handling in the fast path.
For almost all purposes, integer timestamps are preferable to
Mercurial. It stores integer timestamps in the dirstate and would thus
like to avoid doing any float/int comparisons or conversions. We will
continue to have to deal with 1-second granularity on filesystems for
quite some time, so this won't significantly hinder our capabilities.
This has some impact on our file cache validation code in that it
lowers timestamp resolution. But as we still have to deal with
low-resolution filesystems, we're not relying on this anyway.
An alternate approach is to use stat[ST_MTIME], which is guaranteed to
be an integer. But since this support isn't already in our extension,
we can't depend on it being available without adding a hard Python->C
API dependency that's painful for people like yours truly who have
bisect regularly and people without compilers.
The home of 'Abort' is 'error' not 'util' however, a lot of code seems to be
confused about that and gives all the credit to 'util' instead of the
hardworking 'error'. In a spirit of equity, we break the cycle of injustice and
give back to 'error' the respect it deserves. And screw that 'util' poser.
For great justice.
Because st.st_mtime is computed as 'sec + 1e-9 * nsec' and double is too narrow
to represent nanoseconds, int(st.st_mtime) can be 'sec + 1'. Therefore, that
value could be different from the one got by osutils.listdir().
This patch fixes the problem by accessing to raw st_mtime by tuple index.
It catches TypeError to fall back to st.st_mtime because our osutil.stat does
not support tuple index. In dirstate.normal(), 'st' is always a Python stat,
but in dirstate.status(), it can be either a Python stat or an osutil.stat.
Thanks to vgatien-baron@janestreet.com for finding the root cause of this
subtle problem.
Previously, a read(N) where N was less than the length of the first
available chunk would mutate the deque instance twice and allocate a new
str from the slice of the existing chunk. Profiling drawed my attention
to these as a potential hot spot during changegroup reading.
This patch makes the code more complicated in order to avoid the
aforementioned 3 operations.
On a pre-generated mozilla-central gzip bundle, this series has the
following impact on `hg unbundle` performance on my MacBook Pro:
before: 358.21 real 317.69 user 38.49 sys
after: 301.57 real 262.69 user 37.11 sys
delta: -56.64 real -55.00 user -1.38 sys
The new code results in simpler logic within the while loop. It is also
faster since we avoid performing operations on the queue and buf
collections. However, there shouldn't be any super hot loops for this
since the whole point of chunkbuffer is to avoid reading large amounts
of data at once. This does, however, make it easier to optimize
chunkbuffer in a subsequent patch.
