Summary:
We want to allow blocking full repo streaming clones in certain
repositories (since they can take the lock and take a very long time) unless the
client has explicitly asked for it. The existing stream_out wire protocol has no
way of passing an option, so let's create a new endpoint.
Reviewed By: quark-zju
Differential Revision: D7763717
fbshipit-source-id: eace47143f8fdcc4c6e302b5c26678ccf56ca5d4
Summary:
Previously `hg server` uses `HGPORT` that might be in use. This patch uses
`-p 0 --port-file ...` so `hg server` always gets assigned a free port.
The change was first made by the following Ruby script:
```
re = /^ \$ hg serve(.*) -p \$(HGPORT[12]?) (.*[^\\])$\n \$/
Dir['*.t'].each do |path|
old = File.read(path)
new = old.lines.map do |l|
next l if l[/\(glob\)/] or not l['$HGPORT'] or l[/^ [$>]/]
"#{l.chomp} (glob)\n"
end.join.gsub re, <<-'EOS'.chomp
$ hg serve\1 -p 0 --port-file $TESTTMP/.port \3
$ \2=`cat $TESTTMP/.port`
$
EOS
File.write(path, new) if old != new
end
```
Then there are some manual changes:
run-tests.py: It now treats `$HGPORT` in output as glob pattern `*`, since
it does not know the assigned value in tests.
test-bookmarks-pushpull.t, test-https.t: Some `hg pull`s were changed to use
explicit paths instead of relying on `.hgrc` since the test restarts the
server and `.hg/hgrc` having an outdated URL.
test-schemes.t: The test writes `$HGPORT` to `.hgrc` before assigning it.
Changed the order so the correct `$HGPORT` is written.
test-patchbomb-tls.t: Changed `(?) (glob)` to `(glob) (?)`.
Reviewed By: DurhamG
Differential Revision: D6925398
fbshipit-source-id: d5c10476f43ce23f9e99618807580cf8ba92595c
This makes pull consistent with the part used by push and provide us with a
more compact representation of bookmarks.
In addition, this opens the way for smarter bookmark exchanges (e.g. filtering
by names or only sending the bookmark relevant to the pulled set, etc).
Before updating the actual bookmark update, we can start with updating the way
we check for push race. Checking bookmarks state earlier is useful even if we
still use pushkey. Aborting before the changegroup is added can save a lot of
time.
Upon pull or unbundle, we display a message with the range of new revisions
fetched. This revision range could readily be used after a pull to look out
what's new with 'hg log'. The algorithm takes care of filtering "obsolete"
revisions that might be present in transaction's "changes" but should not be
displayed to the end user.
chg only supports 'hg serve' when the options to the serve command
follow the 'hg serve'. For example, 'hg -R <repo> serve ..' is unsupported.
This leads to issues with chg running for the following tests:
- test-bundle2-exchange.t
- test-clone-uncompressed.t
- test-hgweb-csp.t
- test-http-bad-server.t
- test-http-bundle1.t
- test-http-protocol.t
- test-http.t
There was an effort made earlier to fix this issue for chg and the tests were
fixed to confirm to the compatible pattern. But the new tests did not take care
of the same and hence, fail. Hopefully, there will be continuous build setup
for chg after all tests are made compatible with chg so that we can avoid such
issues.
Test Plan:
Ran the aforementioned tests with and without '--chg' option.
Differential Revision: https://phab.mercurial-scm.org/D946
A new bundle2 capability 'phases' has been added. If 'heads' is part of the
supported value for 'phases', the server supports reading and sending 'phase-
heads' bundle2 part.
Server is now able to process a 'phases' boolean parameter to 'getbundle'. If
'True', a 'phase-heads' bundle2 part will be included in the bundle with phase
information relevant to the whole pulled set. If this method is available the
phases listkey namespace will no longer be listed.
Beside the more efficient encoding of the data, this new method will greatly
improve the phase exchange efficiency for repositories with non-served
changesets (obsolete, secret) since we'll no longer send data about the
filtered heads.
Add a new 'devel.legacy.exchange' config item to allow fallback to the old
'listkey in bundle2' method.
Reminder: the pulled set is not just the changesets bundled by the pull. It
also contains changeset selected by the "pull specification" on the client
side (eg: everything for bare pull). One of the reason why the 'pulled set' is
important is to make sure we can move -common- nodes to public.
As the TODO in the test said, the previous error message was not
very helpful. Let's improve things.
Differential Revision: https://phab.mercurial-scm.org/D811
This effectively reverts 43e7ede19a74. Some lines still have (re)
due to variable length port numbers. There's not much we can do
about that. But at least this change removes most of the ugliness.
On system where HGTEST_PORT is configured to a value an order a magnitude lower
or higher than the default. The number of digit in the HGPORT changes and this
test breaks.
This makes the test pass in Python 2.6. It's all one of two problems:
1)
abort: error: '' (2.7)
vs
abort: error: (2.6)
2) 65537 (2.7) vs -1 (2.6)
With some...unfortunate escaping required due to how regular
expressions work. Sigh.
test-http-bad-server has a flaky output case that reproduces fairly
often (20% or more of runs on my Mac) on BSD-family OSes. So that
tests can be green for the 4.2 release, let's more or less give up on
this output, and plan to pursue it in the future.
Many have seen a "stream ended unexpectedly" error. This message is
raised from changegroup.readexactly() when a read(n) operation fails
to return exactly N bytes.
I believe most occurrences of this error in the wild stem from
the code changed in this patch. Before, if bundle2's part applicator
raised an Exception when processing/applying parts, the exception
handler would attempt to iterate the remaining parts. If I/O
during this iteration failed, it would likely raise the
"stream ended unexpectedly" exception.
The problem with this approach is that if we already encountered
an I/O error iterating the bundle2 data during application, then
any further I/O would almost certainly fail. If the original stream
were closed, changegroup.readexactly() would obtain an empty
string, triggering "stream ended unexpectedly" with "got 0." This is
the error message that users would see. What's worse is that the
original I/O related exception would be lost since a new exception
would be raised. This made debugging the actual I/O failure
effectively impossible.
This patch changes the exception handler for bundle2 application to
ignore errors when seeking the underlying stream. When the underlying
error is I/O related, the seek should fail fast and the original
exception will be re-raised. The output changes in
test-http-bad-server.t demonstrate this.
When the underlying error is not I/O related and the stream can be
seeked, the old behavior is preserved.
As part of writing test-http-bad-server.t, I noticed that some
requests include an empty Vary HTTP request header.
The Vary HTTP request header indicates which headers should be taken
into account when determining if a cached response can be used. It also
accepts the special value of "*".
The previous code unconditionally added a Vary header. This could lead
to an empty header value. While I don't believe this violates the HTTP
spec, this is weird and just wastes bytes. So this patch changes
behavior to only send a Vary header when it has a value.
Some low-level wire protocol byte reporting tests changed. In some
cases, the exact point of data termination changed. However, the
behavior being tested - that clients react when the connection is
closed in the middle of an HTTP request line or header - remains
unchanged.
There were a handful of places in the code where HTTPResponse.read()
was called with no explicit error handling or with inconsistent
error handling. In order to eliminate this class of bug, we globally
swap out HTTPResponse.read() with a unified error handler.
I initially attempted to fix all call sites. However, after
going down that rabbit hole, I figured it was best to just change
read() to do what we want. This appears to be a worthwhile
change, as the tests demonstrate many of our uncaught exceptions
go away.
To better represent this class of failure, we introduce a new
error type. The main benefit over IOError is it can hold a hint.
I'm receptive to tweaking its name or inheritance.
I've spent several hours over the past few weeks investigating
networking failures involving hg.mozilla.org. As part of this, it
has become clear that the Mercurial client's error handling when
it encounters network failures is far from robust.
To prove this is true, I've devised a battery of tests simulating
various network failures, notably premature connection closes. To
achieve this, I've implemented an extension that monkeypatches the
built-in HTTP server and hooks in at the socket level and allows
various events to occur based on config options. For example, you
can refuse to accept() a client socket or you can close() the socket
after N bytes have been sent or received. The latter effectively
simulates an unexpected connection drop (and these occur all the
time in the real world).
The new test file launches servers exhibiting various "bad" behaviors
and points a client at them. As the many TODO comments in the test
call attention to, Mercurial often displays unhelpful errors when
network-related failures occur. This makes it difficult for users
to understand what's going on and difficult for server administrators
to pinpoint root causes without packet tracing.
Upcoming patches will attempt to fix these error handling
deficiencies.
