This enables chg to replace a server socket in an atomic way:
1. bind to a temp address
2. listen
3. rename
Currently 3 happens before 2 so a client may see the socket file but fails
to connect to it.
os.fdopen() does not accepts bytes as its second argument which represent the
mode in which the file is to be opened. This patch makes sure unicodes are
passed in py3 by using pycompat.sysstr().
Previously, when a chg server is exiting, it does not handle connected
clients so clients may get ECONNRESET and crash:
1. client connect() # success
2. server shouldexit = True and exit
3. client recv() # ECONNRESET
297d89f2789e makes this race condition easier to reproduce if a lot of short
chg commands are started in parallel.
This patch fixes the above issue by unlinking the socket path to stop
queuing new connections and processing all pending connections before exit.
This patch changes unixforkingservice so it only calls
`self._servicehandler.unlinksocket(self.address)` at most once.
This is needed by the next patch.
This is necessary to solve future dependency cycle between commandserver.py
and chgserver.py.
'cmd' prefix is added to table and function names to avoid conflicts with
hgweb.
Almost all sys.stdin/out/err in hgext/ and mercurial/ are replaced by util's.
There are a few exceptions:
- lsprof.py and statprof.py are untouched since they are a kind of vendor
code and they never import mercurial modules right now.
- ui._readline() needs to replace sys.stdin and stdout to pass them to
raw_input(). We'll need another workaround here.
This makes a channeledoutput thread-safe as long as the underlying fwrite() is
thread-safe. Both POSIX and Windows implementations are documented as MT-safe.
MT-safety is necessary to use ui.fout and ui.ferr in hgweb.
Now setpgid has 2 main purposes: better handling for terminal-generated
SIGTSTP, SIGINT, and process-exit-generated SIGHUP. Update the comment to
explain things more clearly.
The old value 5 was arbitrary chosen. Since there's no practical reason to
limit the backlog, this patch simply uses SOMAXCONN as a value large enough.
SocketServer.ForkingMixIn of Python 2.x has a couple of issues, such as:
- race condition that leads to 100% CPU usage (Python 2.6)
https://bugs.python.org/issue21491
- can't wait for children belonging to different process groups (Python 2.6)
- leaves at least one zombie process (Python 2.6, 2.7)
https://bugs.python.org/issue11109
The first two are critical because we do setpgid(0, 0) in child process to
isolate terminal signals. The last one isn't, but ForkingMixIn seems to be
doing silly. So there are two choices:
a) backport and maintain SocketServer until we can drop support for Python 2.x
b) replace SocketServer by simpler one and eliminate glue codes
I chose (b) because it's great time for getting rid of utterly complicated
SocketServer stuff, and preparing for future move towards prefork service.
New unixforkingservice is implemented loosely based on chg 531f8ef64be6. It
is monolithic but much simpler than SocketServer. unixservicehandler provides
customizing points for chg, and it will be shared with future prefork service.
Old unixservice class is still used by chgserver. It will be removed later.
Thanks to Jun Wu for investigating these issues.
This is common between chg and vanilla forking server, so move it to
commandserver and unify handle().
It would be debatable whether we really need gc.collect() or not, but that
is beyond the scope of this series. Maybe we can remove gc.collect() once
all resource deallocations are switched to context manager.
This will allow us to clear in-memory password storage per runcommand().
I've updated commandserver to call resetstate() of both ui and repo.ui because
they may have different states in theory.
In 'unix' mode, the server is typically detached from the console. Therefore
a client couldn't see the exception that occurred while instantiating the
server object.
This patch tries to catch the early error and send it to 'e' channel even if
the server isn't instantiated yet. This means the error may be sent before the
initial hello message. So it's up to the client implementation whether to
handle the early error message or error out as protocol violation.
The error handling code is also copied to chgserver.py. I'll factor out them
later if we manage to get chg passes the test suite.
These operations are obviously invalid for file-like channels because they
will read or write protocol headers.
This patch works around the issue that "hg archive" generates a corrupted
zip file on Windows commandserver because of unusable tell() implementation.
But the problem still occurs without using a commandserver.
$ hg archive -R not-small-repo -t zip - | cat > invalid.zip
So, this patch cannot fix the issue5049 completely.
A progress bar is normally disabled in command-server session, but chg can
enable it. This patch makes sure the last-print time is measured per command.
Otherwise, progress.delay could be ineffective if a progbar was loaded before
forking worker process.
This patch is corresponding to the following change.
https://bitbucket.org/yuja/chg/commits/2dfe3e90b365
This prepares for porting the chg server. In chg, a server receives client's
stdio over a UNIX domain socket to override server channels. This is because
chg should behave as if it is a normal hg command attached to tty. "nontty"
is not wanted.
This patch is corresponding to the following change. This doesn't test the
identity of "cin" object because the current version of chg reopens stdio
to apply buffering mode.
https://bitbucket.org/yuja/chg/commits/c48c7aed5fc0
Because unknown attributes are delegated to the underlying file object,
commandserver channels said they were '<stdout>' or '<stdin>' even though
they weren't. This patch makes them say '<X-channel>'.
We generally make modules importable from the front-end layer, dispatch ->
commands -> x. So the import cycle to dispatch should be resolved by the
commandserver module.
The home of 'Abort' is 'error' not 'util' however, a lot of code seems to be
confused about that and gives all the credit to 'util' instead of the
hardworking 'error'. In a spirit of equity, we break the cycle of injustice and
give back to 'error' the respect it deserves. And screw that 'util' poser.
For great justice.
Python 2.6 introduced the "except type as instance" syntax, replacing
the "except type, instance" syntax that came before. Python 3 dropped
support for the latter syntax. Since we no longer support Python 2.4 or
2.5, we have no need to continue supporting the "except type, instance".
This patch mass rewrites the exception syntax to be Python 2.6+ and
Python 3 compatible.
This patch was produced by running `2to3 -f except -w -n .`.
Because pipe-mode server uses stdio as IPC channel, other modules should not
touch stdio directly and use ui instead. However, this strategy is brittle
because several Python functions read and write stdio implicitly.
print 'hello' # should use ui.write()
# => ch = 'h', size = 1701604463 'ello', data = '\n'
This patch adds protection for such mistakes. Both stdio files and low-level
file descriptors are redirected to /dev/null while command server uses them.
Because unix-mode server forks child process per connection, client does not
know the pid of the server that will handle requests. The pid is necessary
to interrupt hung process:
1. client connects to socket server
2. server accepts the connection, forks, and tells pid
3. client requests "runcommand pull"
.. hung ..
4. client sends SIGINT to the (forked) server
5. server returns from I/O wait
Note that getsockopt(SO_PEERCRED) of Linux cannot be used because the server
fork()s after accept().
Typical use case of 'unix' mode is a background hg daemon.
$ hg serve --cmdserver unix --cwd / -a /tmp/hg-`id -u`.sock
Unlike 'pipe' mode in which parent process keeps stdio channel, 'unix' server
can be detached. So clients can freely connect and disconnect from server,
saving Python start-up time.
It might be better to write "--cmdserver socket -a unix:/sockpath" instead
of "--cmdserver unix -a /sockpath" in case hgweb gets the ability to listen
on unix domain socket.
This is the stub for new mode that will listen for connections on unix domain
socket.
Though --daemon option is not banned in 'pipe' mode, it is useless because
the detached 'pipe' mode server exits immediately due to null stdin. Should
it abort if --daemon is specified with --cmdserver pipe or --stdio?
The previous patch makes sure that in_ == out, so it's no longer needed
to keep in_ for __getattr__. Also, it seems strange for channeledoutput
to have in_ stream which is actually a writable file object.
It seems wrong to get attributes from object different than the underlying
file. In the following example, it doesn't make sense to flush stderr after
writing to stdout:
self.ferr.write(str(a))
if not getattr(self.ferr, 'closed', False):
self.ferr.flush()
Command server is designed to use the channel protocol even if the server
process is accessible to tty, whereas vanilla hg should be able to read
password from tty in that case. So it isn't enough to swap sys.stdin:
# works only if the server process is detached from the console
sys.stdin = self.fin
getpass.getpass('')
sys.stdin = oldin
or test isatty:
# vanilla hg can't talk to tty if stdin is redirected
if self._isatty(self.fin):
return getpass.getpass('')
else:
...
Since ui.nontty flag is undocumented and command-server channels don't provide
isatty(), this change won't affect the other uses of ui._isatty().
issue3161 also suggests to provide some context of messages. I think it can
be implemented by using the generic templating function.
