urllib.parse.urlencode() returns unicodes on Python 3. This commit adds a
method which will take its output and encode it to bytes so that we can use
bytes consistently.
Mixing bytes and unicode creates a mess. Do things in bytes as possible.
New sysbytes() helper only takes care of ASCII characters, but avoids raising
nasty unicode exception. This is the same design principle as sysstr().
Currently, we export urlparse via util.urlparse then
call util.urlparse.urlparse() and util.urlparse.urlunparse()
in a few places. This is the only url* module exported from
pycompat, making it a one-off. So let's transition to urlreq
to match everything else.
Yes, we double import "urlparse" now on Python 2. This will
be cleaned up in a subsequent patch.
Also, the Python 3 functions trade in str/unicode not bytes.
So we'll likely need to write a custom implementation that
speaks bytes. But moving everyone to an abstracted API
is a good first step.
It is duplicated by urlreq.unquote and is unused. Kill it.
We retain the imports because it is re-exported via util.urlparse,
which is used elsewhere.
Since we no longer access attributes of urlparse at module load time,
this change /should/ result in that module reverting to a lazy module.
Previously, urlreq.unquote aliased to urllib.parse.unquote,
which returned a str/unicode. We like bytes, so switch urlreq.unquote
to dispatch to urllib.parse.unquote_to_bytes.
This required a minor helper function to register an alias under a
different name from which it points. If this turns into a common
pattern, we could likely teach _registeralias to accept tuple
values defining the mapping. Until then, I didn't feel like
adding complexity to _registeralias.
This allows us to handle bytes in mostly the same manner as Python 2 str,
so we can get rid of ugly s[i:i + 1] hacks:
s = bytestr(s)
while i < len(s):
c = s[i]
...
This is the simpler version of the previous RFC patch which tried to preserve
the bytestr type if possible. New version simply drops the bytestr wrapping
so we aren't likely to pass a bytestr to a function that expects Python 3
bytes.
With Python 3.4.3, timeit says 0.437 usec -> 0.0685 usec. With Python
3.6, timeit says 0.157 usec -> 0.0907 usec. So it's faster on both
versions, but the speedup varies a lot.
Thanks to Gregory Szorc for the suggestion.
urllib.parse.quote() accepts either str or bytes and returns str.
There exists a urllib.parse.quote_from_bytes() which only accepts
bytes. We should probably use that to retain strong typing and
avoid surprises.
In addition, since nearly all strings in Mercurial are bytes, we
probably don't want quote() returning unicode.
So, this patch implements a custom quote() that only accepts bytes
and returns bytes. The quoted URL should only contain URL safe
characters which is a strict subset of ASCII. So
`.encode('ascii', 'strict')` should be safe.
urllib.request imports a bunch of symbols from other urllib
modules. We should map to the original symbols not the
re-exported ones because this is more correct. Also, it
will prevent an import of urllib.request if only one of
the lower-level symbols/modules is needed.
pycompat.getenv returns os.getenvb on py3 which is not available on Windows.
This patch replaces them with encoding.environ.get and checks to ensure no
new instances of os.getenv or os.setenv are introduced.
shlex.split() only accepts unicodes on Python 3. After this patch we will be
using pycompat.shlexsplit(). This patch also replaces existing occurences of
shlex.split with pycompat.shlexsplit.
sys.executable on Python 3 returns unicodes and we want bytes. So this patch
adds a new pycompat.sysexecutable which returns bytes by encoding using
os.fsencode() since it is path variable.
os.getenv() on python 3 deals with unicodes. If we want to pass bytes. we have
os.getenvb() which deals with bytes. This patch adds up a pycompat.osgetenv
which deals with bytes on both python 2 and 3.
Keys of keyword arguments need to be str(unicodes) on Python 3. We have a lot
of function where we pass keyword arguments. Having utility functions to help
converting keys to unicodes before passing and convert back them to bytes once
passed into the function will be helpful. We now have functions named
pycompat.strkwargs(dic) and pycompat.byteskwargs(dic) to help us.
getopt.getopt() deals with unicodes on Python 3 internally and if bytes
arguments are passed, then it will return TypeError. So we have now
pycompat.getoptb() which takes bytes arguments, convert them to unicode, call
getopt.getopt() and then convert the returned value back to bytes and then
return those value.
All the instances of getopt.getopt() are replaced with pycompat.getoptb().
Following the behaviour of Python 3, os.getcwd() return unicodes. We need
bytes version as path variables are bytes in UNIX. Python 3 has os.getcwdb()
which returns current working directory in bytes.
Like rest of the things there in pycompat, like osname, ossep, we need to
rewrite every instance of os.getcwd to pycompat.getcwd to make them work
correctly on Python 3.
Since standard streams are TextIO on Python 3, we can't use sys.stdin/out/err
directly. Fortunately we can get the underlying BytesIO via .buffer as long as
the streams aren't replaced by e.g. StringIO.
stdin/out/err are provided through util so we can wrap them by platform API.
open() requires mode argument as unicodes on Python 3. This patch introduces
pycompat.open() which is inserted to files using transformer and replaces
builtins.open() calls.
sys.argv returns unicodes on Python 3. We need a bytes version for us.
There was also a python bug/feature request which wanted then to implement
one. They rejected and it is quoted in one of the comments that we can use
fsencode() to get a bytes version of sys.argv. Though not sure about its
correctness.
Link to the comment: http://bugs.python.org/issue8776#msg217416
After this patch we will have pycompat.sysargv which will return us bytes
version of sys.argv. If this patch goes in, i will like to make transformer
rewrite sys.argv with pycompat.argv because there are lot of occurences.
os.name returns unicodes on py3. Most of our checks are like
os.name == 'nt'
Because of the transformer, on the right hand side we have b'nt'. The
condition will never satisfy even if os.name returns 'nt' as that will be an
unicode.
We either need to encode every occurence of os.name or have a
new variable which is much cleaner. Now we have pycompat.osname.
There are around 53 occurences of os.name in the codebase which needs to
be replaced by pycompat.osname to support Python 3.
The custom module importer was making these bytes, so when we poked
values into self.__dict__ we had bytes instead of unicode on py3 and
it didn't work.
This will be used to convert encoding.encoding to a str acceptable by
Python 3 functions.
The source encoding is changed to "latin-1" because encoding.encoding can
have arbitrary bytes. Since valid names should consist of ASCII characters,
we don't care about the mapping of non-ASCII characters so long as invalid
names are distinct from valid names.
Replacement _pycompatstub designed to be compatible with our demandimporter.
try-except is replaced by version comparison because ImportError will no longer
be raised immediately.
This should be less invasive than mucking builtins.
Since tokenize.untokenize() looks start/end positions of tokens, we calculates
them from the NEWLINE token of the future import.
These functions will be imported automagically by our code transformer.
getattr() and setattr() are widely used in our code. We wouldn't probably
want to rewrite every single call of getattr/setattr. delattr() and hasattr()
aren't that important, but they are functions of the same kind.
We have a single line function which just lowercase the letters and replaces
"_" with "". Its better to avoid that function call. Moreover we calling this
function around 33 times.
pycompat.py includes hack to import modules whose names are changed in Python 3.
We use try-except to load module according to the version of python. But this
method forces us to import the modules to raise an ImportError and hence making
it demandimport unfriendly.
This patch changes the try-except blocks to a single if-else block. To avoid
test-check-pyflakes.t complain about unused imports, pycompat.py is excluded
from the test.
