This lets clients send many getfile requests in a single transaction.
Note that this requires 76fcf62accb0 be applied to your Mercurial, or
you'll be bitten by a bug[0] in Mercurial's wireproto batching. As a
result of this change, remotefilelog now effectively requires the
upcoming Mercurial 3.5 if you want to use a specific release.
0: http://bz.selenic.com/show_bug.cgi?id=4739
Right now, this is a naive fetch-one-file method. The next change will
mark the method as batchable and use a batch in the client so that
many files can be requested in a single RPC.
The way the protocol is defined for getfiles interleaves reading
filenames and sending file contents, which works fine over ssh but is
incompatible with http.
This change is probably not neccessary now that remotefilelog
correctly checks for its own capability first, but it helped me debug
so I left it in for completeness.
If we instead wrap wireproto.capabilities, then our capabilities don't
get transmitted via the hello command, so not all clients will notice
the new capability unless we do the wrapping here.
Test output is in the test that previously demonstrated the
defect. Note that there's still a defect: we're advertising the
capability over http even though we have no hope of the getfiles
method working over http.
The magic string 'internal' causes Mercurial to never blame
remotefilelog for being broken. I had suspected that remotefilelog
might work with 3.4, but the tests fail against 3.4.1, so I'm just
making testedwith empty.
The rev graph building code was flawed because it didn't track second parents
correctly. This was caught when someone was developing an extension and
attempted to commit a merge commit in some way.
repo.sopener has been deprecated since hg 2.3, and repo.svfs replaces
it. Since it's been dead for so long, let's just use svfs and call it
good enough.
Summary:
The incominghook was meant to pregenerate any remotefilelog blobs that were
likely to be needed shortly. Unfortunately it actually just slows down pushes,
since in large repos the hook takes longer than the push does sometimes.
So let's just remove it.
Test Plan: Apparently there were no tests for this :p
Reviewers: sid0, lcharignon, mitrandir, ericsumner, rmcelroy
Reviewed By: rmcelroy
Differential Revision: https://phabricator.fb.com/D2185894
Signature: t1:2185894:1435126819:e1e1125520411356eccff4baee31ab2938ebc0fe
Summary: I really don't think it should be in this list.
Test Plan: `hg`
Reviewers: durham, #sourcecontrol, rmcelroy
Reviewed By: durham, #sourcecontrol, rmcelroy
Subscribers: rmcelroy
Differential Revision: https://phabricator.fb.com/D1997655
Signature: t1:1997655:1429189594:aa8f355a6fc61e300f824be6b2fbd64a42dde2b5
Summary:
When adjustlinkrevs got moved to the filectx upstream, we incorrectly
moved it to the remotefilectx inside remotefilelog. We don't actually use
remotefilectx on the server, so wrapping it did nothing.
The fix is to move the wrapping to be in remotefilelogserver.py so it is
executed on the server side.
Test Plan:
Did a checkout with my shallow client pointed at a full repo with no
blob cache. Verified it went quickly (minutes, instead of hours).
Reviewers: pyd
Differential Revision: https://phabricator.fb.com/D2097851
Summary:
Since we only prefetch things that are in the sparse checkout, copy tracing
(which touches everything in the manifest diff) would do individual file
downloads for every file. Let's just remove those files from the copy tracing
check entirely since the user probably doesn't care if they're outside the
sparse checkout.
Test Plan: Added a test
Reviewers: sid0, rmcelroy, lcharignon, pyd
Differential Revision: https://phabricator.fb.com/D2083768
Summary:
Match with with latest version of core to pass the test.
There were a couple of changes in core that broke the extension, I matched
those changes to make the test pass.
Test Plan: The tests are all passing
Reviewers: durham
Differential Revision: https://phabricator.fb.com/D2053958
Upstream now has a matcher on _computeforwardmissing which will allow us to only
prefetch the necessary parts of a sparse checkout.
Since we're now being returned an iterator, we need to convert it to a list
since we iterate over it and return it.
Summary:
Previously remotefilelog would prefetch every file in a commit. With the sparse
checkout extension we want to only prefetch things in the sparse checkout.
This commit makes remotefilelog aware of the possible existence of a sparse
matcher.
Test Plan: Added tests
Reviewers: sid0, rmcelroy, pyd, lcharignon
Subscribers: kang
Differential Revision: https://phabricator.fb.com/D1967207
Summary:
Per @pyd's review of D1933267, we need to check for the linknode in cl.nodemap,
not in cl (whose __contains__ method only looks for revs and doesn't even check
for visibility... lolz).
Test Plan: ran tests
Reviewers: durham, sid0, pyd, ericsumner, lcharignon, davidsp, mitrandir
Reviewed By: mitrandir
Subscribers: akushner, daviser, pyd
Differential Revision: https://phabricator.fb.com/D1934941
Tasks: 6573011
Signature: t1:1934941:1427130649:b084635db9bfcd28c4d4a1bcf12a7500c06b323c
Summary:
The new version of adjust linknodes wasn't accounting for the fact that some
ancestries contained nodes that no longer exist. Check for that before looking
for common ancestors.
The old version of this code survived by luck. We were catching KeyErrors as one
base case, and it just happens that LookupError from the changelog is also a
KeyError, so it was getting caught and eaten.
Test Plan:
We should probably add a test, but I have to leave shortly and this is pretty
broken, so we'll have to take a rain check.
Reviewers: rmcelroy, pyd, sid0
Differential Revision: https://phabricator.fb.com/D1933267
Summary:
The new fixmappinglinknodes function was using recursion to traverse the file
history, but this would break for files with history that was extremely long
(stack overflow). Switch to using a manual stack approach.
Test Plan: Ran the tests (I'd added a test to cover this logic before).
Reviewers: sid0, davidsp, mitrandir, lcharignon, pyd, rmcelroy
Reviewed By: rmcelroy
Subscribers: michaelbarton
Differential Revision: https://phabricator.fb.com/D1931944
Signature: t1:1931944:1426884986:3a0ef144fb55b8c0533e5c5de90699a1823b891f
Summary: I'm going to add a new parameter upstream. Make this more generic so that we don't have to try and support both the old and the new versions.
Test Plan: Ran tests with both old and new hg.
Reviewers: davidsp, rmcelroy, akushner, pyd, daviser, mitrandir, ericsumner, durham
Reviewed By: durham
Differential Revision: https://phabricator.fb.com/D1920172
Signature: t1:1920172:1426615175:d90bda3b3cc30f6e5f3149af82ae9e43dee39455
Summary:
Previously remotefilelog did not produce all the necessary local data blobs
when doing a peer push/pull if the incoming changegroup had two manifests
that referred to the same file revision. We would only create a file blob
containing the history for the first occurrence, then if the user tried to
access the file history for other occurrences they got an exception.
The fix is to add linkrev fixup logic, similar to the adjustlinkrev() method
from core Mercurial's filectx. Now, if no valid local file blob can be found, we
will compute a valid history by reading the changelog.
We might be able to write this data to disk in the future as well to prevent
having to repeatedly compute this.
Test Plan: Added a test
Reviewers: sid0, rmcelroy, pyd, mitrandir, lcharignon
Differential Revision: https://phabricator.fb.com/D1904453
Summary:
For hg-git conversions we're going to cause commits without actually updating to the base. Currently, this will cause lots of individual fetches.
The test demonstrates the issue -- wihtout this patch it'll fetch the 2 files over 2 fetches, but with it it'll fetch the files over 1 fetch.
Test Plan: Ran the tests.
Reviewers: davidsp, rmcelroy, akushner, pyd, daviser, mitrandir, ericsumner, durham
Reviewed By: durham
Differential Revision: https://phabricator.fb.com/D1893721
Tasks: 6390769
Signature: t1:1893721:1425624679:5651f71d5023919e9321646275b681b573847c44
Upstream has refactored the copy logic to compute the file lists in separate
functions, so we no longer need to compute the file lists ourselves.
Update the README's Mercurial min-version since this change depends on new APIs
inside Mercurial.
Summary:
Upstream has moved _adjustlinkrev from being a global function to one
on the filectx. Let's do the same.
Test Plan: Ran the tests
Reviewers: mitrandir
Differential Revision: https://phabricator.fb.com/D1825043
Summary:
adjustlinkrev makes ancestor reading orders of magnitude slower,
so we need to avoid using it. Since adjustlinkrev already returns the linkrev in
certain cases, let's just force it to always return that during file blob
creation.
Test Plan:
Generated a few thousand blobs for www and fbcode using the old and new
methods and verified that they were byte-for-byte identical.
Reviewers: sid0, pyd, mpm, rmcelroy
Differential Revision: https://phabricator.fb.com/D1782400
Summary:
If the remotefilelog server was not specified in the hgrc, or if the project
hgrc wasn't trusted, it would throw an obtuse error about a NoneType string.
This fixes it to give a more informative error explaining the problem.
Test Plan: Added a test
Reviewers: sid0, pyd, mitrandir, ericsumner, rmcelroy
Reviewed By: rmcelroy
Differential Revision: https://phabricator.fb.com/D1774743
Signature: t1:1774743:1420830544:5122a8e11f668ee8c35996e0f4395883a31ce8b0
Summary:
There are reports of the local cache becoming invalid when stored on disk. This
adds an option that will do some basic validation and remediation for those
entries, and log some data to disk.
This is optional, since it incurs some performance overhead. We just want to use
it long enough to track down the issue.
Test Plan: Added a test
Reviewers: sid0, pyd, ericsumner, rmcelroy, mitrandir
Reviewed By: mitrandir
Differential Revision: https://phabricator.fb.com/D1774724
Signature: t1:1774724:1420827432:06ace9d1dc078f469e0f61ebd7f604fc3b606f6d
Summary:
We've gotten reports of corrupt cache files, and the error message is pretty
obtuse (ValueError for converting a string to an int). This refactors the size
check into a function and provides a better error message.
Test Plan: Added a test
Reviewers: sid0, pyd, mitrandir, ericsumner, rmcelroy
Reviewed By: rmcelroy
Differential Revision: https://phabricator.fb.com/D1774721
Signature: t1:1774721:1420830671:afd54dde8fdc00e08ed1c6cb73bf9fdc7fac2327
Summary: We were forgetting to pass these arguments on to the child function.
Test Plan: Visual inspection.
Reviewers: durham, davidsp, rmcelroy, akushner, pyd, daviser, mitrandir, ericsumner
Reviewed By: ericsumner
Differential Revision: https://phabricator.fb.com/D1773782
Signature: t1:1773782:1420765574:d73be08ab25265e4769d8bf70671f2ea1c13f8dd
Mercurial upstream does some fancy stuff inside introrev now to provide the
correct introrev. It relies on having the filelog though, so we need to avoid
it. Remotefilelog has perfect history knowledge, so we can just return the
correct linkrev.
Summary:
We're seeing some weird cache corruption errors when writing the cache to disk.
My best bet is there's multiple writes colliding and causing bad data, so let's
do atomic renames.
Test Plan: Ran the test suite
Reviewers: sid0, pyd, davidsp, rmcelroy
Reviewed By: rmcelroy
Subscribers: ericsumner, mitrandir
Differential Revision: https://phabricator.fb.com/D1747190
Signature: t1:1747190:1418865586:0a07e5243dfe9c1d5ea24f81874910d1080f24e2
It is part of the revlog API and some extension like tortoisehg rely on it. The
default implementation is the same as size so we can safely mimic this here.
A recent fix to make ancestor maps work with changeset evolution actually caused
a pretty serious regression. The ancestormap validation code was returning
ancestormaps with hidden ancestors if the first commit in the history was a
hidden node. This resulted in lots of invalid ancestories being returned.
Instead we only want to allow hidden ancestors in the map if the relativeto
commit has been explicitly set to a hidden node.
Summary: Last bits needed to get remotefilelog over bundle2 working. Includes tests.
Test Plan: Ran tests, including with `--extra-config-opt experimental.bundle2-exp=True`
Reviewers: davidsp, akushner, pyd, rmcelroy, daviser, durham
Reviewed By: durham
Differential Revision: https://phabricator.fb.com/D1671738
Tasks: 5568731
Signature: t1:1671738:1415676482:b9e7a1f308919526b0c41fee54d89da876518ec7
Certain filectx constructions used the rev number of the self._changeid. We
need to convert that to a node before using it. This was breaking blame. I've
now added a blame test too.
Bundlerepos work by providing a fake revlog layer above an existing revlog.
Since remotefilelog doesn't use revlogs for filelogs, bundlerepo's did not work.
This commit fixes it such that you can now hg pull from a bundle, as long as
that bundle is shallow (i.e. contains no file contents). This will work for the
common use case of trying to recover data from .hg/strip-backups.
For reference, shallow bundles don't contain any file data because we never
delete any file data from .hg/store/data when using remotefilelog. Even after
the commits have been stripped.
Upstream Mercurial has moved localrepo.pull into exchange.pull. This moves our
wrapping of that command out of shallowrepo and into __init__. Exchange is
becoming an increasingly important class, so we may want to think about moving
all exchange wrapper logic out to a separate module in remotefilelog.
