Summary:
The incominghook was meant to pregenerate any remotefilelog blobs that were
likely to be needed shortly. Unfortunately it actually just slows down pushes,
since in large repos the hook takes longer than the push does sometimes.
So let's just remove it.
Test Plan: Apparently there were no tests for this :p
Reviewers: sid0, lcharignon, mitrandir, ericsumner, rmcelroy
Reviewed By: rmcelroy
Differential Revision: https://phabricator.fb.com/D2185894
Signature: t1:2185894:1435126819:e1e1125520411356eccff4baee31ab2938ebc0fe
Summary: I really don't think it should be in this list.
Test Plan: `hg`
Reviewers: durham, #sourcecontrol, rmcelroy
Reviewed By: durham, #sourcecontrol, rmcelroy
Subscribers: rmcelroy
Differential Revision: https://phabricator.fb.com/D1997655
Signature: t1:1997655:1429189594:aa8f355a6fc61e300f824be6b2fbd64a42dde2b5
Summary:
When adjustlinkrevs got moved to the filectx upstream, we incorrectly
moved it to the remotefilectx inside remotefilelog. We don't actually use
remotefilectx on the server, so wrapping it did nothing.
The fix is to move the wrapping to be in remotefilelogserver.py so it is
executed on the server side.
Test Plan:
Did a checkout with my shallow client pointed at a full repo with no
blob cache. Verified it went quickly (minutes, instead of hours).
Reviewers: pyd
Differential Revision: https://phabricator.fb.com/D2097851
Summary:
Since we only prefetch things that are in the sparse checkout, copy tracing
(which touches everything in the manifest diff) would do individual file
downloads for every file. Let's just remove those files from the copy tracing
check entirely since the user probably doesn't care if they're outside the
sparse checkout.
Test Plan: Added a test
Reviewers: sid0, rmcelroy, lcharignon, pyd
Differential Revision: https://phabricator.fb.com/D2083768
Summary:
Match with with latest version of core to pass the test.
There were a couple of changes in core that broke the extension, I matched
those changes to make the test pass.
Test Plan: The tests are all passing
Reviewers: durham
Differential Revision: https://phabricator.fb.com/D2053958
Upstream now has a matcher on _computeforwardmissing which will allow us to only
prefetch the necessary parts of a sparse checkout.
Since we're now being returned an iterator, we need to convert it to a list
since we iterate over it and return it.
Summary:
Previously remotefilelog would prefetch every file in a commit. With the sparse
checkout extension we want to only prefetch things in the sparse checkout.
This commit makes remotefilelog aware of the possible existence of a sparse
matcher.
Test Plan: Added tests
Reviewers: sid0, rmcelroy, pyd, lcharignon
Subscribers: kang
Differential Revision: https://phabricator.fb.com/D1967207
Summary:
Per @pyd's review of D1933267, we need to check for the linknode in cl.nodemap,
not in cl (whose __contains__ method only looks for revs and doesn't even check
for visibility... lolz).
Test Plan: ran tests
Reviewers: durham, sid0, pyd, ericsumner, lcharignon, davidsp, mitrandir
Reviewed By: mitrandir
Subscribers: akushner, daviser, pyd
Differential Revision: https://phabricator.fb.com/D1934941
Tasks: 6573011
Signature: t1:1934941:1427130649:b084635db9bfcd28c4d4a1bcf12a7500c06b323c
Summary:
The new version of adjust linknodes wasn't accounting for the fact that some
ancestries contained nodes that no longer exist. Check for that before looking
for common ancestors.
The old version of this code survived by luck. We were catching KeyErrors as one
base case, and it just happens that LookupError from the changelog is also a
KeyError, so it was getting caught and eaten.
Test Plan:
We should probably add a test, but I have to leave shortly and this is pretty
broken, so we'll have to take a rain check.
Reviewers: rmcelroy, pyd, sid0
Differential Revision: https://phabricator.fb.com/D1933267
Summary:
The new fixmappinglinknodes function was using recursion to traverse the file
history, but this would break for files with history that was extremely long
(stack overflow). Switch to using a manual stack approach.
Test Plan: Ran the tests (I'd added a test to cover this logic before).
Reviewers: sid0, davidsp, mitrandir, lcharignon, pyd, rmcelroy
Reviewed By: rmcelroy
Subscribers: michaelbarton
Differential Revision: https://phabricator.fb.com/D1931944
Signature: t1:1931944:1426884986:3a0ef144fb55b8c0533e5c5de90699a1823b891f
Summary: I'm going to add a new parameter upstream. Make this more generic so that we don't have to try and support both the old and the new versions.
Test Plan: Ran tests with both old and new hg.
Reviewers: davidsp, rmcelroy, akushner, pyd, daviser, mitrandir, ericsumner, durham
Reviewed By: durham
Differential Revision: https://phabricator.fb.com/D1920172
Signature: t1:1920172:1426615175:d90bda3b3cc30f6e5f3149af82ae9e43dee39455
Summary:
Previously remotefilelog did not produce all the necessary local data blobs
when doing a peer push/pull if the incoming changegroup had two manifests
that referred to the same file revision. We would only create a file blob
containing the history for the first occurrence, then if the user tried to
access the file history for other occurrences they got an exception.
The fix is to add linkrev fixup logic, similar to the adjustlinkrev() method
from core Mercurial's filectx. Now, if no valid local file blob can be found, we
will compute a valid history by reading the changelog.
We might be able to write this data to disk in the future as well to prevent
having to repeatedly compute this.
Test Plan: Added a test
Reviewers: sid0, rmcelroy, pyd, mitrandir, lcharignon
Differential Revision: https://phabricator.fb.com/D1904453
Summary:
For hg-git conversions we're going to cause commits without actually updating to the base. Currently, this will cause lots of individual fetches.
The test demonstrates the issue -- wihtout this patch it'll fetch the 2 files over 2 fetches, but with it it'll fetch the files over 1 fetch.
Test Plan: Ran the tests.
Reviewers: davidsp, rmcelroy, akushner, pyd, daviser, mitrandir, ericsumner, durham
Reviewed By: durham
Differential Revision: https://phabricator.fb.com/D1893721
Tasks: 6390769
Signature: t1:1893721:1425624679:5651f71d5023919e9321646275b681b573847c44
Upstream has refactored the copy logic to compute the file lists in separate
functions, so we no longer need to compute the file lists ourselves.
Update the README's Mercurial min-version since this change depends on new APIs
inside Mercurial.
Summary:
Upstream has moved _adjustlinkrev from being a global function to one
on the filectx. Let's do the same.
Test Plan: Ran the tests
Reviewers: mitrandir
Differential Revision: https://phabricator.fb.com/D1825043
Summary:
adjustlinkrev makes ancestor reading orders of magnitude slower,
so we need to avoid using it. Since adjustlinkrev already returns the linkrev in
certain cases, let's just force it to always return that during file blob
creation.
Test Plan:
Generated a few thousand blobs for www and fbcode using the old and new
methods and verified that they were byte-for-byte identical.
Reviewers: sid0, pyd, mpm, rmcelroy
Differential Revision: https://phabricator.fb.com/D1782400
Summary:
If the remotefilelog server was not specified in the hgrc, or if the project
hgrc wasn't trusted, it would throw an obtuse error about a NoneType string.
This fixes it to give a more informative error explaining the problem.
Test Plan: Added a test
Reviewers: sid0, pyd, mitrandir, ericsumner, rmcelroy
Reviewed By: rmcelroy
Differential Revision: https://phabricator.fb.com/D1774743
Signature: t1:1774743:1420830544:5122a8e11f668ee8c35996e0f4395883a31ce8b0
Summary:
There are reports of the local cache becoming invalid when stored on disk. This
adds an option that will do some basic validation and remediation for those
entries, and log some data to disk.
This is optional, since it incurs some performance overhead. We just want to use
it long enough to track down the issue.
Test Plan: Added a test
Reviewers: sid0, pyd, ericsumner, rmcelroy, mitrandir
Reviewed By: mitrandir
Differential Revision: https://phabricator.fb.com/D1774724
Signature: t1:1774724:1420827432:06ace9d1dc078f469e0f61ebd7f604fc3b606f6d
Summary:
We've gotten reports of corrupt cache files, and the error message is pretty
obtuse (ValueError for converting a string to an int). This refactors the size
check into a function and provides a better error message.
Test Plan: Added a test
Reviewers: sid0, pyd, mitrandir, ericsumner, rmcelroy
Reviewed By: rmcelroy
Differential Revision: https://phabricator.fb.com/D1774721
Signature: t1:1774721:1420830671:afd54dde8fdc00e08ed1c6cb73bf9fdc7fac2327
There's a slight different in core Mercurial between a blame where the file was
modified in the current commit, versus a file that was modified in an older
commit. This difference affects remotefilelog (since it uses linkrevs to look
back for the first commit), so fix the test to cover this scenario.
Summary: We were forgetting to pass these arguments on to the child function.
Test Plan: Visual inspection.
Reviewers: durham, davidsp, rmcelroy, akushner, pyd, daviser, mitrandir, ericsumner
Reviewed By: ericsumner
Differential Revision: https://phabricator.fb.com/D1773782
Signature: t1:1773782:1420765574:d73be08ab25265e4769d8bf70671f2ea1c13f8dd
Mercurial upstream does some fancy stuff inside introrev now to provide the
correct introrev. It relies on having the filelog though, so we need to avoid
it. Remotefilelog has perfect history knowledge, so we can just return the
correct linkrev.
Summary:
We're seeing some weird cache corruption errors when writing the cache to disk.
My best bet is there's multiple writes colliding and causing bad data, so let's
do atomic renames.
Test Plan: Ran the test suite
Reviewers: sid0, pyd, davidsp, rmcelroy
Reviewed By: rmcelroy
Subscribers: ericsumner, mitrandir
Differential Revision: https://phabricator.fb.com/D1747190
Signature: t1:1747190:1418865586:0a07e5243dfe9c1d5ea24f81874910d1080f24e2
It is part of the revlog API and some extension like tortoisehg rely on it. The
default implementation is the same as size so we can safely mimic this here.
A recent fix to make ancestor maps work with changeset evolution actually caused
a pretty serious regression. The ancestormap validation code was returning
ancestormaps with hidden ancestors if the first commit in the history was a
hidden node. This resulted in lots of invalid ancestories being returned.
Instead we only want to allow hidden ancestors in the map if the relativeto
commit has been explicitly set to a hidden node.
a989a083752e in upstream mercurial changed which deltas get chosen for revlogs,
which affects the size of the data being transfered. So we just update the test
accordingly.
Summary: Last bits needed to get remotefilelog over bundle2 working. Includes tests.
Test Plan: Ran tests, including with `--extra-config-opt experimental.bundle2-exp=True`
Reviewers: davidsp, akushner, pyd, rmcelroy, daviser, durham
Reviewed By: durham
Differential Revision: https://phabricator.fb.com/D1671738
Tasks: 5568731
Signature: t1:1671738:1415676482:b9e7a1f308919526b0c41fee54d89da876518ec7
Summary:
- fixes for pulls after uncompressed clones
- add a glob where there should be one
Test Plan: Ran the tests with Mercurial @
Reviewers: davidsp, akushner, rmcelroy, pyd, daviser, durham
Reviewed By: durham
Differential Revision: https://phabricator.fb.com/D1668739
Signature: t1:1668739:1415643384:567a0841ef4367ee2f905f6fe5517d229d5eadcf
Certain filectx constructions used the rev number of the self._changeid. We
need to convert that to a node before using it. This was breaking blame. I've
now added a blame test too.
Bundlerepos work by providing a fake revlog layer above an existing revlog.
Since remotefilelog doesn't use revlogs for filelogs, bundlerepo's did not work.
This commit fixes it such that you can now hg pull from a bundle, as long as
that bundle is shallow (i.e. contains no file contents). This will work for the
common use case of trying to recover data from .hg/strip-backups.
For reference, shallow bundles don't contain any file data because we never
delete any file data from .hg/store/data when using remotefilelog. Even after
the commits have been stripped.
Upstream Mercurial has moved localrepo.pull into exchange.pull. This moves our
wrapping of that command out of shallowrepo and into __init__. Exchange is
becoming an increasingly important class, so we may want to think about moving
all exchange wrapper logic out to a separate module in remotefilelog.
repo.revs() no longer returns an object that can be indexed, so we can't use []
on it anymore. So let's use list() on it first.
The bookmark output from upstream Mercurial has also changed, so we need to
update the tests.
Summary:
When doing 'hg unshelve foo.txt' with Changeset Evolution enabled, uncommit will
first prune the commit, then try to read the filelog history to determine if any
renames need to be undone. Since the commit is now pruned, remotefilelog fails
to find any valid histories.
This fixes it two allow hidden histories if the filectx commit is hidden. It
also tweaks remotefilectx to produce commit-relative histories when possible,
which will result in more accurate histories.
Test Plan:
Ran hg uncommit in the evolve repo that had problems before. Verified
it now worked.
Reviewers: pyd, sid0
Differential Revision: https://phabricator.fb.com/D1587306
Summary:
Pull-prefetch would not download file versions from the server if the file
version already existed in the local cache or the local store data.
Unfortunately, if someone landed their commit, then later stripped their local
version, the local store data file version might become invalid and no local
cache version would exist. Meaning things like 'commit' might fail when offline.
This changes prefetch to always fetch from the server when dealing with files it
knows are from revs on the server.
Test Plan:
Added a test that makes local commits that already exist on the
server, and verifies that a pull-prefetch fetches the server file version,
despite that same version existing locally.
Reviewers: sid0, pyd, davidsp
Subscribers: orip
Differential Revision: https://phabricator.fb.com/D1607260
Summary:
The ResponseError exception expect a second argument. Otherwise the code
handling it crashes.
Test Plan: The handling of the response error stop crashing.
Reviewers: durham
Differential Revision: https://phabricator.fb.com/D1581574
Summary:
If the orig function crash before the fileservice is installed, the finally
clause explode, shadowing the original error. We fixes thats.
Test Plan:
crash stopped being shadowed but crash in the finally clause.
Reviewers: durham
Differential Revision: https://phabricator.fb.com/D1581562
Summary: API change
Test Plan: @durham ran an amend.
Reviewers: durham
Reviewed By: durham
Subscribers: durham
Differential Revision: https://phabricator.fb.com/D1569510
Summary:
Upstream Mercurial changed the way merging works and added
revlog.commonancestorsheads. This changes remotefilelog to implement the same
API.
Previously we were able to use ancestors.genericancestors to do the graph
traversal. Upstream Mercurial has deleted that function though (since it is now
unused), so remotefilelog must now build a temporary rev graph in order to use
the ancestors.* apis.
Test Plan: Added a test. It failed without the fix, it passes with the fix.
Reviewers: sid0, davidsp, pyd
Differential Revision: https://phabricator.fb.com/D1566787
