On AIX, ksh builtin printf does not understand \NNN. Some tests use this
to generate test data, and so fail on AIX. Rework these tests to use python
to generate the correct characters. This fixes the tests on AIX and should
be more generally portable.
Changeset 6c87ffe1c40d is testing how a EUC-JP encoded Japanese file
is treated in different encodings, so the ISO-8859-1 encoded German
file is no longer needed.
This patch treats all files inside repository as encoded by
locale's encoding when pygmentize.
We can assume that most files are written in locale's encoding,
but current implementation treats them as UTF-8.
So there's no way to specify the encoding of files.
Current implementation, e2f3244d5179 (issue1341):
1. Convert original `text`, which is treated as UTF-8, to locale's encoding.
`encoding.tolocal()` is the method to convert from internal UTF-8 to local.
If original `text` is not UTF-8, e.g. Japanese EUC-JP, some characters
become garbled here.
2. pygmentize, with no UnicodeDecodeError.
This patch:
1. Convert original `text`, which is treated as locale's encoding, to unicode.
Pygments prefers unicode object than raw str. [1]_
If original `text` is not encoded by locale's encoding, some characters
become garbled here.
2. pygmentize, also with no UnicodeDecodeError :)
3. Convert unicode back to raw str, which is encoded by locale's.
.. [1] http://pygments.org/docs/unicode/
The test copied get-with-headers.py from $TESTDIR and committed it to
a test repository. The test output therefore depended unnecessarily on
the exact content of get-with-headers.py. It has now been replaced
with another small Python script.
Recent pygments guess AntlrJavaLexer for the isolatin text, converted to ascii
by util.tolocal, which yields class="err". Guess lexer by extension instead.
Example case:
Display file written in iso-8859-1 with current HGENCODING utf-8.
At the moment only an Error page appears because pygmentize
chokes on the replacement chars.
Alternatives:
1) Turn off highlighting and avoid UnicodeDecodeError
for files that are not in HGENCODING.
2) [this patch] use util.tolocal to display these files.
Alternative 2) seems ok, as this only concerns display and
readability.
See also: c5f1a58b8b9a, apparently put aside during refactor of
highlight.
Add test for UnicodeDecodeError with iso-8859-1 file contents.
For non-html mimetypes it doesn't make much sense. This also fixes the
issue that highlight unconditionally adds a <link/> tag for its CSS to
the template's header (which is pointless in text/plain output).
This patch allows a per-repository (for example, within a hgwebdir) selection
of pygments_style web option. No static .css files required.
Test edited by pmezard (compatibility fixes)
