Unfortunately, the ssh and http implementations are slightly different
due to differences in their _callstream implementations, which
prevents ssh from behaving streamily. We should probably introduce a
new batch command that can stream results over ssh at some point in
the near future.
The streamy behavior of batch over http(s) is an enormous win for
remotefilelog over http: in my testing, it's saving about 40% on file
fetches with a cold cache against a server on localhost.
The home of 'Abort' is 'error' not 'util' however, a lot of code seems to be
confused about that and gives all the credit to 'util' instead of the
hardworking 'error'. In a spirit of equity, we break the cycle of injustice and
give back to 'error' the respect it deserves. And screw that 'util' poser.
For great justice.
The output pipe does not have manually managed read buffer (actually, no read
anything). To also use the doublepipe for outgoing write (useful to consume
remote output when pushing large set of data) we need the doublepipe to work on
standard pipe too.
This class is responsible for ensuring we still process the server output
streamed through the ssh's 'stderr' pipe during the initial wait for other
protocol streams.
It currently only works on posix system because of its use of 'select.select'.
This is necessary to use non-blocking IO base on polling. Such polling is
needed to restore real time output with ssh peer.
Changeset 6a565336bae3 is talking about 5x regression on Mac OS X when playing
with this value. So we introduced our own buffering layer in previous
changesets. This seems to keep the regression away (we are even issuing much
less read).
We need this pipe to still be buffered when will switch to unbuffered pipe.
(switch motivated by the need of using polling to restore real time output from
ssh server). This is the only pipe that needs to be wrapped because this is the
one who do extensive usage of 'readline'. The stderr pipe of the process is
alway read in non blocking raw chunk, so it won't benefit from the
buffering.
When we'll be using the ssh's 'stderr' for realtime output successfully, it will
no longer be possible to use 'stderr' to carry the error message (because it
is already consumed by the real time output logic.
This feature is very rarely used (test says largefile only) and I think
breaking its output pattern is worth the benefit of having real time output with
ssh.
Reading all available data from a pipe has a platform-dependent
implementation.
This patch establishes platform.readpipe() by copying the
inline implementation in sshpeer.readerr(). The implementations
for POSIX and Windows are currently identical. The POSIX
implementation will be changed in a subsequent patch.
This implements the call needed by bundle2. The error handling is a bit flaky
right now, but bundle2 error handling in general is still flaky anyway. Bundle2
is still disabled by default, I do not expect this to be a problem.
We already have multiple call function for multiple return type. The
`_decompress` function is only used for http and seems like a layer violation.
We drop it in favor of a new call type dedicated to "stream that may be useful to
compress".
Previously, if another command was run with --verbose, and for whatever reason
that invoked sshpeer, we'd get a 'running ssh' message from sshpeer. This extra
line would interfere with that command's output and cause dumb parsers to
break.
For example, hg annotate can be run with --verbose to get full usernames. This,
combined with the third-party remotefilelog extension which can cause ssh
connections to be created, leads to an extra 'running ssh' line that breaks
most parsers.
This patch is (BC) because hg pull --verbose will no longer print out exactly
what ssh command it is running.
No tests are affected by this change.
When running 'hg pull --rebase', I was seeing this exception 100% of the
time as the python process was closing down:
Exception TypeError: TypeError("'NoneType' object is not callable",) in
<bound method Popen.__del__ of <subprocess.Popen object at 0x937c10>> ignored
By storing the subprocess on the sshpeer, the subprocess seems to clean up
correctly, and I no longer see the exception. I have no idea why this actually
works, but I get a 0% repro if I store the subprocess in self.subprocess,
and a 100% repro if I store None in self.subprocess.
Possibly related to issue 2240.
This change separates peer implementations from the repository implementation.
localpeer currently is a simple pass-through to localrepository, except for
legacy calls, which have already been removed from localpeer. This ensures that
the local client code only uses the most modern peer API when talking to local
repos.
Peers have a .local() method which returns either None or the underlying
localrepository (or descendant thereof). Repos have a .peer() method to return
a freshly constructed localpeer. The latter is used by hg.peer(), and also to
allow folks to pass either a peer or a repo to some generic helper methods.
We might want to get rid of .peer() eventually.
The only user of locallegacypeer is debugdiscovery, which uses it to pose as a
pre-setdiscovery client. But we decided to leave the old API defined in
locallegacypeer for clarity and maybe for other uses in the future.
It might be nice to actually define the peer API directly in peer.py as stub
methods. One problem there is, however, that localpeer implements
lock/addchangegroup, whereas the true remote peers implement unbundle.
It might be desireable to get rid of this distinction eventually.
