mirror of
https://github.com/facebook/sapling.git
synced 2024-10-11 17:27:53 +03:00
d678fe1702
Summary: `ascii` was used as the default / fallback, which is not a user-friendly choice. Nowadays utf-8 dominates: - Rust stdlib is utf-8. - Ruby since 1.9 is utf-8 by default. - Python 3 is unicode by default. - Windows 10 adds utf-8 code page. Given the fact that: - Our CI sets HGENCODING to utf-8 - Nuclide passes `--encoding=utf-8` to every command. - Some people have messed up with `LC_*` and complained about hg crashes. - utf-8 is a super set of ascii, nobody complains that they want `ascii` encoding and the `utf-8` encoding messed their setup up. Let's just use `utf-8` as the default encoding. More aggressively, if someone sets `ascii` as the encoding, it's almost always a mistake. Auto-correct that to `utf-8` too. This should also make future integration with Rust easier (where it's enforced utf-8 and does not have an option to change the encoding). In the future we might just drop the flexibility of choosing customized encoding, so this diff autofixes `ascii` to `utf-8`, instead of allowing `ascii` to be set. We cannot enforce `utf-8` yet, because of Windows. Here is our encoding strategy vs the upstream's: | item | upstream | | ours | ours | | | current | ideal | current | ideal | | CLI argv | bytes | bytes | utf-8 [1] | utf-8 | | path | bytes | auto [3] | migrating [2] | utf-8 | | commit message | utf-8 | utf-8 | utf-8 | utf-8 | | bookmark name | utf-8 | utf-8 | utf-8 | utf-8 | | file content | bytes | bytes | bytes | bytes | [1]: Argv was accidentally enforced utf-8 for command-line arguments by a Rust wrapper. But it simplified a lot of things and is kind of ok: everything that can be passed as CLI arguments are utf-8: -M commit message, -b bookmark, paths, etc. There is no "file content" passed via CLI arguments. [2]: Path is controversial, because it's possible for systems to have non-utf8 paths. The upstream behavior is incorrect if a repo gets shared among different encoding systems (ex. both Linux and Windows). We have to know the encoding of paths to be able to convert them suitable for the local system. One way is to enforce UTF-8 for paths. The other is to keep encoding information stored with individual paths (like Ruby strings). The UTF-8 approach is much simpler with the tradeoff that non-utf-8 paths become unsupported, which seems to be a reasonable trade-off. [3]: See https://www.mercurial-scm.org/wiki/WindowsUTF8Plan. Reviewed By: singhsrb Differential Revision: D17098991 fbshipit-source-id: c0ff1e586a887233bd43cdb854fb3538aa9b70c2
191 lines
4.7 KiB
Perl
191 lines
4.7 KiB
Perl
Test character encoding
|
|
|
|
$ hg init t
|
|
$ cd t
|
|
|
|
we need a repo with some legacy latin-1 changesets
|
|
|
|
$ hg unbundle "$TESTDIR/bundles/legacy-encoding.hg"
|
|
adding changesets
|
|
adding manifests
|
|
adding file changes
|
|
added 2 changesets with 2 changes to 1 files
|
|
new changesets 1e78a93102a3:0e5b7e3f9c4a
|
|
$ hg co
|
|
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
|
|
$ $PYTHON << EOF
|
|
> f = file('latin-1', 'w'); f.write("latin-1 e' encoded: \xe9"); f.close()
|
|
> f = file('utf-8', 'w'); f.write("utf-8 e' encoded: \xc3\xa9"); f.close()
|
|
> f = file('latin-1-tag', 'w'); f.write("\xe9"); f.close()
|
|
> EOF
|
|
|
|
should fail with encoding error
|
|
|
|
$ echo "plain old ascii" > a
|
|
$ hg st
|
|
M a
|
|
? latin-1
|
|
? latin-1-tag
|
|
? utf-8
|
|
$ HGENCODING=ascii hg ci -l latin-1
|
|
transaction abort!
|
|
rollback completed
|
|
abort: decoding near ' encoded: \xe9': 'utf8' codec can't decode byte 0xe9 in position 20: unexpected end of data! (esc)
|
|
[255]
|
|
|
|
these should work
|
|
|
|
$ echo "latin-1" > a
|
|
$ HGENCODING=latin-1 hg ci -l latin-1
|
|
$ echo "utf-8" > a
|
|
$ HGENCODING=utf-8 hg ci -l utf-8
|
|
|
|
hg log (ascii)
|
|
|
|
$ hg --encoding ascii log
|
|
changeset: 3:ca661e7520de
|
|
tag: tip
|
|
user: test
|
|
date: Thu Jan 01 00:00:00 1970 +0000
|
|
summary: utf-8 e' encoded: ?
|
|
|
|
changeset: 2:650c6f3d55dd
|
|
user: test
|
|
date: Thu Jan 01 00:00:00 1970 +0000
|
|
summary: latin-1 e' encoded: ?
|
|
|
|
changeset: 1:0e5b7e3f9c4a
|
|
user: test
|
|
date: Mon Jan 12 13:46:40 1970 +0000
|
|
summary: koi8-r: ????? = u'\u0440\u0442\u0443\u0442\u044c'
|
|
|
|
changeset: 0:1e78a93102a3
|
|
user: test
|
|
date: Mon Jan 12 13:46:40 1970 +0000
|
|
summary: latin-1 e': ? = u'\xe9'
|
|
|
|
|
|
hg log (latin-1)
|
|
|
|
$ hg --encoding latin-1 log
|
|
changeset: 3:ca661e7520de
|
|
tag: tip
|
|
user: test
|
|
date: Thu Jan 01 00:00:00 1970 +0000
|
|
summary: utf-8 e' encoded: \xe9 (esc)
|
|
|
|
changeset: 2:650c6f3d55dd
|
|
user: test
|
|
date: Thu Jan 01 00:00:00 1970 +0000
|
|
summary: latin-1 e' encoded: \xe9 (esc)
|
|
|
|
changeset: 1:0e5b7e3f9c4a
|
|
user: test
|
|
date: Mon Jan 12 13:46:40 1970 +0000
|
|
summary: koi8-r: \xd2\xd4\xd5\xd4\xd8 = u'\\u0440\\u0442\\u0443\\u0442\\u044c' (esc)
|
|
|
|
changeset: 0:1e78a93102a3
|
|
user: test
|
|
date: Mon Jan 12 13:46:40 1970 +0000
|
|
summary: latin-1 e': \xe9 = u'\\xe9' (esc)
|
|
|
|
|
|
hg log (utf-8)
|
|
|
|
$ hg --encoding utf-8 log
|
|
changeset: 3:ca661e7520de
|
|
tag: tip
|
|
user: test
|
|
date: Thu Jan 01 00:00:00 1970 +0000
|
|
summary: utf-8 e' encoded: \xc3\xa9 (esc)
|
|
|
|
changeset: 2:650c6f3d55dd
|
|
user: test
|
|
date: Thu Jan 01 00:00:00 1970 +0000
|
|
summary: latin-1 e' encoded: \xc3\xa9 (esc)
|
|
|
|
changeset: 1:0e5b7e3f9c4a
|
|
user: test
|
|
date: Mon Jan 12 13:46:40 1970 +0000
|
|
summary: koi8-r: \xc3\x92\xc3\x94\xc3\x95\xc3\x94\xc3\x98 = u'\\u0440\\u0442\\u0443\\u0442\\u044c' (esc)
|
|
|
|
changeset: 0:1e78a93102a3
|
|
user: test
|
|
date: Mon Jan 12 13:46:40 1970 +0000
|
|
summary: latin-1 e': \xc3\xa9 = u'\\xe9' (esc)
|
|
|
|
|
|
hg tags (ascii)
|
|
|
|
$ HGENCODING=ascii hg tags
|
|
tip 3:ca661e7520de
|
|
|
|
hg tags (latin-1)
|
|
|
|
$ HGENCODING=latin-1 hg tags
|
|
tip 3:ca661e7520de
|
|
|
|
hg tags (utf-8)
|
|
|
|
$ HGENCODING=utf-8 hg tags
|
|
tip 3:ca661e7520de
|
|
|
|
hg tags (JSON)
|
|
|
|
$ hg tags -Tjson
|
|
[
|
|
{
|
|
"node": "ca661e7520dec3f5438a63590c350bebadb04989",
|
|
"rev": 3,
|
|
"tag": "tip",
|
|
"type": ""
|
|
}
|
|
]
|
|
|
|
hg log (utf-8)
|
|
|
|
$ HGENCODING=utf-8 hg log
|
|
changeset: 3:ca661e7520de
|
|
tag: tip
|
|
user: test
|
|
date: Thu Jan 01 00:00:00 1970 +0000
|
|
summary: utf-8 e' encoded: \xc3\xa9 (esc)
|
|
|
|
changeset: 2:650c6f3d55dd
|
|
user: test
|
|
date: Thu Jan 01 00:00:00 1970 +0000
|
|
summary: latin-1 e' encoded: \xc3\xa9 (esc)
|
|
|
|
changeset: 1:0e5b7e3f9c4a
|
|
user: test
|
|
date: Mon Jan 12 13:46:40 1970 +0000
|
|
summary: koi8-r: \xc3\x92\xc3\x94\xc3\x95\xc3\x94\xc3\x98 = u'\\u0440\\u0442\\u0443\\u0442\\u044c' (esc)
|
|
|
|
changeset: 0:1e78a93102a3
|
|
user: test
|
|
date: Mon Jan 12 13:46:40 1970 +0000
|
|
summary: latin-1 e': \xc3\xa9 = u'\\xe9' (esc)
|
|
|
|
|
|
hg log (dolphin)
|
|
|
|
$ HGENCODING=dolphin hg log
|
|
abort: unknown encoding: dolphin
|
|
(please check your locale settings)
|
|
[255]
|
|
$ cp latin-1-tag .hg/branch
|
|
$ HGENCODING=latin-1 hg ci -m 'auto-promote legacy name'
|
|
|
|
$ cd ..
|
|
|
|
Test roundtrip encoding/decoding of utf8b for generated data
|
|
|
|
#if hypothesis
|
|
|
|
>>> from hypothesishelpers import *
|
|
>>> from edenscm.mercurial import encoding
|
|
>>> roundtrips(st.binary(), encoding.fromutf8b, encoding.toutf8b)
|
|
Round trip OK
|
|
|
|
#endif
|