This patch removes the global variable "allowhardlinks" that disables
hardlink in all cases, so hardlink gets enabled if the filesystem type is
whitelisted.
Third party extensions wanting to enable hardlink support unconditionally
can replace "_hardlinkfswhitelist.__contains__".
Since osutil.getfstype is available, use it to detect filesystem types. The
whitelist currently includes common local filesystems on Linux where they
should have good hardlink support. We may add new filesystems for other
platforms later.
pycompat.urlreq.unquote and pycompat.urlunquote effectively alias the
same thing. pycompat.urlunquote is only used once in the code base.
So let's switch to urlreq.unquote.
"Effectively" in the above paragraph is because pycompat.urlreq.unquote
aliases urllib.unquote and pycompat.urlunquote aliases urlparse.unquote
on Python 2. You might think one of urllib.unquote and urlparse.unquote
is an alias to the other, but you would be incorrect. In fact, these
functions are copies of each other. There is even a comment in the
CPython source code saying to keep them in sync. You can't make this
up.
Throughout mercurial cdoe, there is a common pattern of attempting to remove
a file and ignoring ENOENT errors. Let's move this into a common function to
allow for cleaner code.
Previously, there were two slightly different versions of unlinkpath between
windows and posix, but these differences were eliminated in previous patches.
Now we can unify these two code paths inside of the util module.
Changeset 3b9cdb72931f removed the mutable default value, but did not explicitly
tested for None. Such implicit checking can introduce semantic and performance
issue. We move to an explicit check for None as recommended by PEP8:
https://www.python.org/dev/peps/pep-0008/#programming-recommendations
string_escape doesn't exist on Python 3, but fortunately the undocumented
codecs.escape_encode() function exists on CPython 2.6, 2.7, 3.5 and PyPy 5.6.
So let's use it for now.
http://stackoverflow.com/a/23151714
It was specified to be an empty list in 3d8abfdaa08a in 2007.
It was correct at the time. But when the function was
refactored in 7bca0f2718ab (2010), it started expecting a dict.
I guess this code path is untested?
Thanks to Yuya for spotting this.
I don't think this is any tight loops and we'd need to worry about
PyObject creation overhead. Also, I'm pretty sure strptime()
will be much slower than PyObject creation (date parsing is
surprisingly slow).
To enable extensions to enable hardlinks for certain environments, let's move
the 'if False' to be an 'if allowhardlinks' and let extensions modify the
allowhardlinks variable.
Tests on linux ext4 pass with it set to True and to False.
As documented for timeit.default_timer, there are better timers available for
performance measures on some platforms. These timers don't have a set epoch,
and thus are only useful for interval measurements, but have higher
resolution, and thus get you a better measurement overall.
Use the same selection logic as Python's timeit.default_timer. This is a
platform clock on Python 2 and early Python 3, and time.perf_counter on Python
3.3 and later (where time.perf_counter is introduced as the best timer to use).
pager replaced stdout with a line buffered version to work around glibc
deciding on a buffering strategy on the first write to stdout. This is going
to make my next patch hard, as replacing stdout will make tracking time
spent blocked on it more challenging.
Move the line buffering requirement to util.py, and remove it from pager.
This means that the abuse of ui.formatted=True and pager set to cat or equivalent
no longer results in a line-buffered output to a pipe, hence (BC), although
I don't expect anyone to be affected
dict.keys() is documented to return a copy, so it's surprising that
sortdict.keys() did not. I noticed this because we have an extension
that calls readlocaltags(). That method tries to remove any tags that
point to non-existent revisions (most likely stripped). However, since
it's unintentionally working on the instance it's modifying, it
sometimes fails to remove tags when there are multiple bad tags in a
row. This was not caught because localrepo.tags() does an additional
layer of filtering.
sortdict is also used in other places, but I have not checked whether
its keys() and/or __delitem__() methods are used there.
util.buffer() either returns inbuilt buffer function or defines a new one which
slices. The inbuilt buffer() also has a length argument which is missing from
the ones we defined. This patch adds that length argument.
Previously, compression engines had APIs for performing revlog
compression but no mechanism to perform revlog decompression. This
patch changes that.
Revlog decompression is slightly more complicated than compression
because in the compression case there is (currently) only a single
engine that can be used at a time. However for decompression, a
revlog could contain chunks from multiple compression engines. This
means decompression needs to map to multiple engines and
decompressors. This functionality is outside the scope of this patch.
But it drives the decision for engines to declare a byte header
sequence that identifies revlog data as belonging to an engine and
an API for obtaining an engine from a revlog header.
As part of "zstd all of the things," we need to teach revlogs to
use non-zlib compression formats. Because we're routing all compression
via the "compression manager" and "compression engine" APIs, we need to
introduction functionality there for performing revlog operations.
Ideally, revlog compression and decompression operations would be
implemented in terms of simple "compress" and "decompress" primitives.
However, there are a few considerations that make us want to have a
specialized primitive for handling revlogs:
1) Performance. Revlogs tend to do compression and especially
decompression operations in batches. Any overhead for e.g.
instantiating a "context" for performing an operation can be
noticed. For this reason, our "revlog compressor" primitive is
reusable. For zstd, we reuse the same compression "context" for
multiple operations. I've measured this to have a performance
impact versus constructing new contexts for each operation.
2) Specialization. By having a primitive dedicated to revlog use,
we can make revlog-specific choices and leave the door open for
more functionality in the future. For example, the zstd revlog
compressor may one day make use of dictionary compression.
A future patch will introduce a decompress() on the compressor
object.
The code for the zlib compressor is basically copied from
revlog.compress(). Although it doesn't handle the empty input
case, the null first byte case, and the 'u' prefix case. These
cases will continue to be handled in revlog.py once that code is
ported to use this API.
The 'author' and 'desc' revsets are documented to be case insensitive.
Unfortunately, this was implemented in 'author' by forcing the input to
lowercase, including for regex like '\B'. (This actually inverts the meaning of
the sequence.) For backward compatibility, we will keep that a case insensitive
regex, but by using matcher options instead of brute force.
This doesn't preclude future hypothetical 'icase-literal:' style prefixes that
can be provided by the user. Such user specified cases can probably be handled
up front by stripping 'icase-', setting the variable, and letting it drop
through the existing code.
This patch implements a new compression engine API allowing
compression engines to declare support for the wire protocol.
Support is declared by returning a compression format string
identifier that will be added to payloads to signal the compression
type of data that follows and default integer priorities of the
engine.
Accessor methods have been added to the compression engine manager
class to facilitate use.
Note that the "none" and "bz2" engines declare wire protocol support
but aren't enabled by default due to their priorities being 0. It
is essentially free from a coding perspective to support these
compression formats, so we do it in case anyone may derive use from
it.
Add the ability for revlog objects to process revision flags and apply
registered transforms on read/write operations.
This patch introduces:
- the 'revlog._processflags()' method that looks at revision flags and applies
flag processors registered on them. Due to the need to handle non-commutative
operations, flag transforms are applied in stable order but the order in which
the transforms are applied is reversed between read and write operations.
- the 'addflagprocessor()' method allowing to register processors on flags.
Flag processors are defined as a 3-tuple of (read, write, raw) functions to be
applied depending on the operation being performed.
- an update on 'revlog.addrevision()' behavior. The current flagprocessor design
relies on extensions to wrap around 'addrevision()' to set flags on revision
data, and on the flagprocessor to perform the actual transformation of its
contents. In the lfs case, this means we need to process flags before we meet
the 2GB size check, leading to performing some operations before it happens:
- if flags are set on the revision data, we assume some extensions might be
modifying the contents using the flag processor next, and we compute the
node for the original revision data (still allowing extension to override
the node by wrapping around 'addrevision()').
- we then invoke the flag processor to apply registered transforms (in lfs's
case, drastically reducing the size of large blobs).
- finally, we proceed with the 2GB size check.
Note: In the case a cachedelta is passed to 'addrevision()' and we detect the
flag processor modified the revision data, we chose to trust the flag processor
and drop the cachedelta.
os.name returns unicodes on py3 and we have pycompat.osname which returns
bytes. This series of 2 patches will change every ocurrence of os.name with
pycompat.osname.
