According to Apple Technote 1150 (unavailable from Apple as far as I
can tell, but archived in several places online), HFS+ ignores sixteen
specific unicode runes when doing path normalization. We need to
handle those cases, so this function lets us efficiently strip the
offending characters from a UTF-8 encoded string (which is the only
way it seems to matter on OS X.)
39fbe33f95fa brought "asciilower" and "import parsers" into
"encoding.py".
This works fine with "parsers" module in C implementation, but doesn't
with one in pure Python implementation, because the latter causes
cyclic dependency below and aborting execution:
util => i18n => encoding => parsers => util
This patch delays importing "parsers" module until it is really
needed, to avoid cyclic dependency around "parsers" in pure Python
build.
This benefits, among other things, the case collision auditor.
On a Linux system with a large real-world repo where all filenames are ASCII,
hg perfcca:
before: wall 0.260157 comb 0.270000 user 0.230000 sys 0.040000 (best of 38)
after: wall 0.164616 comb 0.160000 user 0.160000 sys 0.000000 (best of 54)
We need a way to efficiently lowercase ASCII strings. For example, 'hg status'
needs to build up the fold map -- a map from a canonical case (for OS X,
lowercase) to the actual case of each file and directory in the dirstate.
The current way we do that is to try decoding to ASCII and then calling
lower() on the string, labeled 'orig' below:
str.decode('ascii')
return str.lower()
This is pretty inefficient, and it turns out we can do much better.
I also tested out a condition-based approach, labeled 'cond' below:
(c >= 'A' && c <= 'Z') ? (c + ('a' - 'A')) : c
'cond' turned out to be slower in all cases. A 256-byte lookup table with
invalid values for everything past 127 performed similarly, but this was less
verbose.
On OS X 10.9 with LLVM version 6.0 (clang-600.0.51), the asciilower function
was run against two corpuses.
Corpus 1 (list of files from real-world repo, > 100k files):
orig: wall 0.428567 comb 0.430000 user 0.430000 sys 0.000000 (best of 24)
cond: wall 0.077204 comb 0.070000 user 0.070000 sys 0.000000 (best of 100)
lookup: wall 0.060714 comb 0.060000 user 0.060000 sys 0.000000 (best of 100)
Corpus 2 (mozilla-central, 113k files):
orig: wall 0.238406 comb 0.240000 user 0.240000 sys 0.000000 (best of 42)
cond: wall 0.040779 comb 0.040000 user 0.040000 sys 0.000000 (best of 100)
lookup: wall 0.037623 comb 0.040000 user 0.040000 sys 0.000000 (best of 100)
On a Linux server-class machine with GCC 4.4.6 20120305 (Red Hat 4.4.6-4):
Corpus 1 (real-world repo, > 100k files):
orig: wall 0.260899 comb 0.260000 user 0.260000 sys 0.000000 (best of 38)
cond: wall 0.054818 comb 0.060000 user 0.060000 sys 0.000000 (best of 100)
lookup: wall 0.048489 comb 0.050000 user 0.050000 sys 0.000000 (best of 100)
Corpus 2 (mozilla-central, 113k files):
orig: wall 0.153082 comb 0.150000 user 0.150000 sys 0.000000 (best of 65)
cond: wall 0.031007 comb 0.040000 user 0.040000 sys 0.000000 (best of 100)
lookup: wall 0.028793 comb 0.030000 user 0.030000 sys 0.000000 (best of 100)
SSE instructions might help even more, but I didn't experiment with those.
Newly added 'trim' is used to trim multi-byte characters at most
specified columns correctly: directly slicing byte sequence should be
replaced with 'encoding.trim', because the former may split at
intermediate multi-byte sequence.
Slicing unicode sequence ('uslice') and concatenation with ellipsis
('concat') are defined as function, to make enhancement in subsequent
patch easier.
This copies the performance hack from encoding.lower (e7a5733d533f).
The case-folding logic that kicks in on case-insensitive filesystems
hits encoding.upper hard: with a repository with 75k files, the
timings went from
hg perfstatus
! wall 3.156000 comb 3.156250 user 1.625000 sys 1.531250 (best of 3)
to
hg perfstatus
! wall 2.390000 comb 2.390625 user 1.078125 sys 1.312500 (best of 5)
This is a 24% decrease. For comparison, Mercurial 2.0 gives:
hg perfstatus
! wall 2.172000 comb 2.171875 user 0.984375 sys 1.187500 (best of 5)
so we're only 10% slower than before we added the extra case-folding
logic.
The same decrease is seen when executing 'hg status' as normal, where
we go from:
hg status --time
time: real 4.322 secs (user 2.219+0.000 sys 2.094+0.000)
to
hg status --time
time: real 3.307 secs (user 1.750+0.000 sys 1.547+0.000)
When calling encode on a str, the string is first decoded using the
default encoding and then encoded. So
s.encode('ascii') == s.decode().encode('ascii')
We don't care about the encode step here -- we're just after the
UnicodeDecodeError raised by decode if it finds a non-ASCII character.
This way is also marginally faster since it saves the construction of
the extra str object.
If the default python encoding was changed from ascii, the attempt to
encode as ascii before lower() could throw a UnicodeEncodeError.
Catch UnicodeError instead to prevent an unhandled exception.
this patch uses encoding.lower/upper for case folding, because ones of
str can not fold case of non ascii characters correctly.
to avoid cyclic dependency and to encapsulate logic of normcase in
each platforms, this patch introduces encodinglower/encodingupper in
both posix/windows specific files.
this patch does not change implementation of normcase() in posix.py,
because we do not know the encoding of filenames on POSIX.
some "normcase()" are excluded from function wrap list in
hgext/win32mbcs.py, because they become encoding aware by this patch.
neither number of 'bytes' in any encoding nor 'characters' is
appropriate to calculate terminal columns for specified string.
this patch modifies MBTextWrapper for:
- overriding '_wrap_chunks()' to make it use not built-in 'len()'
but 'encoding.colwidth()' for columns of string
- fixing '_cutdown()' to make it use 'encoding.colwidth()' instead
of local, similar but incorrect implementation
this patch also modifies 'encoding.py':
- dividing 'colwith()' into 2 pieces: one for calculation columns of
specified UNICODE string, and another for rest part of original
one. the former is used from MBTextWrapper in 'util.py'.
- preventing 'colwidth()' from evaluating HGENCODINGAMBIGUOUS
configuration per each invocation: 'unicodedata.east_asian_width'
checking is kept intact for reducing startup cost.
localstr's hash method exists to prevent bogus matching on lossy local
encodings. For instance, we don't want 'caf?' to match 'café' in an
ASCII locale.
But when café can be losslessly encoded in the local charset, we can
simply use a normal string and avoid the hashing trick.
This avoids using localstr's hash method, which would prevent a match between
The current implementation of colwidth was treating 'A'mbiguous
characters as wide, which was incorrect in a non-East Asian context.
As per http://unicode.org/reports/tr11/#Recommendations, we should
instead default to 'narrow' if we don't know better. As character
width is dependent on the particular font used and we have no idea
what fonts are in use, this recommendation applies.
This introduces HGENCODINGAMBIGUOUS to get the old behavior back.
Prior to version 2.7, calling locale.getpreferredencoding() would
always return 'mac-roman' on Mac OS X. Previously, this was handled by
a call to locale.setlocale(). Unfortunately, Python 2.6.5 and older
have a bug where isspace() would incorrectly report True for 0x85 and
0xa0 after such a call.
In order to fix this, we replace the previous _encodingfixup mapping
to an _encodingfixers mapping. Rather than mapping encodings to their
replacement, it maps them to a function returning the
replacement. This allows us to provide an simplified implementation of
getpreferredencoding() which extracts the expected encoding and
restores the locale.
This fix is based on a patch originally submitted by Martijn Pieters
as well as feedback from Brodie Rao.
Mercurial has problem around text wrapping/filling in MBCS encoding
environment, because standard 'textwrap' module of Python can not
treat it correctly. It splits byte sequence for one character into two
lines.
According to unicode specification, "east asian width" classifies
characters into:
W(ide), N(arrow), F(ull-width), H(alf-width), A(mbiguous)
W/N/F/H can be always recognized as 2/1/2/1 bytes in byte sequence,
but 'A' can not. Size of 'A' depends on language in which it is used.
Unicode specification says:
If the context(= language) cannot be established reliably they
should be treated as narrow characters by default
but many of class 'A' characters are full-width, at least, in Japanese
environment.
So, this patch treats class 'A' characters as full-width always for
safety wrapping.
This patch focuses only on MBCS safe-ness, not on writing/printing
rule strict wrapping for each languages
MBCS sensitive textwrap class is originally implemented
by ITO Nobuaki <daydream.trippers@gmail.com>.
