We were not prefetching the potential dependent files for the filelog revisions
we received over the wire. This resulted in a lot of non-batched downloads,
which was super slow. This fixes it by batch downloading the parents and delta
parents of the incoming filelog revisions and adds a test.
touch -t is portable, but requires some computation to get a date
value that's a week ago. A Python oneliner is a little goofy, but
seemed like a straightforward enough answer that I chose that.
Right now, this is a naive fetch-one-file method. The next change will
mark the method as batchable and use a batch in the client so that
many files can be requested in a single RPC.
If we instead wrap wireproto.capabilities, then our capabilities don't
get transmitted via the hello command, so not all clients will notice
the new capability unless we do the wrapping here.
Test output is in the test that previously demonstrated the
defect. Note that there's still a defect: we're advertising the
capability over http even though we have no hope of the getfiles
method working over http.
1) The client doesn't look-before-you-leap on the remotefilelog capability
2) The http server crashes ungracefully when handling a remotefilelog
request (ideally it should respond with '400 No Such Method' as a
server without the extension would.)
3) capabilities are inconsistently advertised between cmd=hello and
cmd=capabilities.
Future patches will attempt to clean up most of this.
Summary:
Since we only prefetch things that are in the sparse checkout, copy tracing
(which touches everything in the manifest diff) would do individual file
downloads for every file. Let's just remove those files from the copy tracing
check entirely since the user probably doesn't care if they're outside the
sparse checkout.
Test Plan: Added a test
Reviewers: sid0, rmcelroy, lcharignon, pyd
Differential Revision: https://phabricator.fb.com/D2083768
Summary:
Previously remotefilelog would prefetch every file in a commit. With the sparse
checkout extension we want to only prefetch things in the sparse checkout.
This commit makes remotefilelog aware of the possible existence of a sparse
matcher.
Test Plan: Added tests
Reviewers: sid0, rmcelroy, pyd, lcharignon
Subscribers: kang
Differential Revision: https://phabricator.fb.com/D1967207
Summary:
Per @pyd's review of D1933267, we need to check for the linknode in cl.nodemap,
not in cl (whose __contains__ method only looks for revs and doesn't even check
for visibility... lolz).
Test Plan: ran tests
Reviewers: durham, sid0, pyd, ericsumner, lcharignon, davidsp, mitrandir
Reviewed By: mitrandir
Subscribers: akushner, daviser, pyd
Differential Revision: https://phabricator.fb.com/D1934941
Tasks: 6573011
Signature: t1:1934941:1427130649:b084635db9bfcd28c4d4a1bcf12a7500c06b323c
Summary:
Previously remotefilelog did not produce all the necessary local data blobs
when doing a peer push/pull if the incoming changegroup had two manifests
that referred to the same file revision. We would only create a file blob
containing the history for the first occurrence, then if the user tried to
access the file history for other occurrences they got an exception.
The fix is to add linkrev fixup logic, similar to the adjustlinkrev() method
from core Mercurial's filectx. Now, if no valid local file blob can be found, we
will compute a valid history by reading the changelog.
We might be able to write this data to disk in the future as well to prevent
having to repeatedly compute this.
Test Plan: Added a test
Reviewers: sid0, rmcelroy, pyd, mitrandir, lcharignon
Differential Revision: https://phabricator.fb.com/D1904453
Summary:
For hg-git conversions we're going to cause commits without actually updating to the base. Currently, this will cause lots of individual fetches.
The test demonstrates the issue -- wihtout this patch it'll fetch the 2 files over 2 fetches, but with it it'll fetch the files over 1 fetch.
Test Plan: Ran the tests.
Reviewers: davidsp, rmcelroy, akushner, pyd, daviser, mitrandir, ericsumner, durham
Reviewed By: durham
Differential Revision: https://phabricator.fb.com/D1893721
Tasks: 6390769
Signature: t1:1893721:1425624679:5651f71d5023919e9321646275b681b573847c44
Summary:
If the remotefilelog server was not specified in the hgrc, or if the project
hgrc wasn't trusted, it would throw an obtuse error about a NoneType string.
This fixes it to give a more informative error explaining the problem.
Test Plan: Added a test
Reviewers: sid0, pyd, mitrandir, ericsumner, rmcelroy
Reviewed By: rmcelroy
Differential Revision: https://phabricator.fb.com/D1774743
Signature: t1:1774743:1420830544:5122a8e11f668ee8c35996e0f4395883a31ce8b0
Summary:
There are reports of the local cache becoming invalid when stored on disk. This
adds an option that will do some basic validation and remediation for those
entries, and log some data to disk.
This is optional, since it incurs some performance overhead. We just want to use
it long enough to track down the issue.
Test Plan: Added a test
Reviewers: sid0, pyd, ericsumner, rmcelroy, mitrandir
Reviewed By: mitrandir
Differential Revision: https://phabricator.fb.com/D1774724
Signature: t1:1774724:1420827432:06ace9d1dc078f469e0f61ebd7f604fc3b606f6d
Summary:
We've gotten reports of corrupt cache files, and the error message is pretty
obtuse (ValueError for converting a string to an int). This refactors the size
check into a function and provides a better error message.
Test Plan: Added a test
Reviewers: sid0, pyd, mitrandir, ericsumner, rmcelroy
Reviewed By: rmcelroy
Differential Revision: https://phabricator.fb.com/D1774721
Signature: t1:1774721:1420830671:afd54dde8fdc00e08ed1c6cb73bf9fdc7fac2327
There's a slight different in core Mercurial between a blame where the file was
modified in the current commit, versus a file that was modified in an older
commit. This difference affects remotefilelog (since it uses linkrevs to look
back for the first commit), so fix the test to cover this scenario.
a989a083752e in upstream mercurial changed which deltas get chosen for revlogs,
which affects the size of the data being transfered. So we just update the test
accordingly.
Summary: Last bits needed to get remotefilelog over bundle2 working. Includes tests.
Test Plan: Ran tests, including with `--extra-config-opt experimental.bundle2-exp=True`
Reviewers: davidsp, akushner, pyd, rmcelroy, daviser, durham
Reviewed By: durham
Differential Revision: https://phabricator.fb.com/D1671738
Tasks: 5568731
Signature: t1:1671738:1415676482:b9e7a1f308919526b0c41fee54d89da876518ec7
Summary:
- fixes for pulls after uncompressed clones
- add a glob where there should be one
Test Plan: Ran the tests with Mercurial @
Reviewers: davidsp, akushner, rmcelroy, pyd, daviser, durham
Reviewed By: durham
Differential Revision: https://phabricator.fb.com/D1668739
Signature: t1:1668739:1415643384:567a0841ef4367ee2f905f6fe5517d229d5eadcf
Certain filectx constructions used the rev number of the self._changeid. We
need to convert that to a node before using it. This was breaking blame. I've
now added a blame test too.
Bundlerepos work by providing a fake revlog layer above an existing revlog.
Since remotefilelog doesn't use revlogs for filelogs, bundlerepo's did not work.
This commit fixes it such that you can now hg pull from a bundle, as long as
that bundle is shallow (i.e. contains no file contents). This will work for the
common use case of trying to recover data from .hg/strip-backups.
For reference, shallow bundles don't contain any file data because we never
delete any file data from .hg/store/data when using remotefilelog. Even after
the commits have been stripped.
repo.revs() no longer returns an object that can be indexed, so we can't use []
on it anymore. So let's use list() on it first.
The bookmark output from upstream Mercurial has also changed, so we need to
update the tests.
Summary:
Pull-prefetch would not download file versions from the server if the file
version already existed in the local cache or the local store data.
Unfortunately, if someone landed their commit, then later stripped their local
version, the local store data file version might become invalid and no local
cache version would exist. Meaning things like 'commit' might fail when offline.
This changes prefetch to always fetch from the server when dealing with files it
knows are from revs on the server.
Test Plan:
Added a test that makes local commits that already exist on the
server, and verifies that a pull-prefetch fetches the server file version,
despite that same version existing locally.
Reviewers: sid0, pyd, davidsp
Subscribers: orip
Differential Revision: https://phabricator.fb.com/D1607260
Summary:
Upstream Mercurial changed the way merging works and added
revlog.commonancestorsheads. This changes remotefilelog to implement the same
API.
Previously we were able to use ancestors.genericancestors to do the graph
traversal. Upstream Mercurial has deleted that function though (since it is now
unused), so remotefilelog must now build a temporary rev graph in order to use
the ancestors.* apis.
Test Plan: Added a test. It failed without the fix, it passes with the fix.
Reviewers: sid0, davidsp, pyd
Differential Revision: https://phabricator.fb.com/D1566787
Summary:
With recent version of mercurial (>= 3.2, 4dfcf21a6aa7), revert uses status
information to determine the files that needs to be touched. It then offer a
simple handle for extensions that needs prefetch.
Test Plan:
Ran the tests. Certain tests depended on the old revert behavior (of
prefetching everything), so they required slight changes.
Reviewers: pyd, sid0, davidsp
Differential Revision: https://phabricator.fb.com/D1551059
Summary:
Changegroups have been refactored upstream and we need to update our
remotefilelog monkey patching accordingly.
Also fix an issue with the tests where 'function foo()' was not considered valid
on certain systems.
Test Plan: Ran the tests
Reviewers: pyd, sid0, davidsp
Differential Revision: https://phabricator.fb.com/D1551019
Summary:
Previously, if pullprefetch was set, we'd perform a prefetch of the
entire manifest of the specified revs (usually the public bookmarks). This
involved stat-ing all the relevant files in the cache to see if they already
existed, which added an extra 6 seconds or so to every pull.
Now we only prefetch the files that are different from our working copy. We
assume we already have all the files that are in our working copy. This reduces
the pullprefetch overhead significantly.
Test Plan:
Did a pull on my laptop. Verified it didn't hang for 6 seconds at the
prefetch stage. Also updated a test
Reviewers: davidsp, pyd, sid0
Reviewed By: sid0
Differential Revision: https://phabricator.fb.com/D1505841
Tasks: 4608894
Summary:
Previously, pullprefetch was executed during the repo.pull stage. This happens
before the bookmarks have been moved, so revsets like 'bookmark()' would
prefetch the wrong commits.
This change moves the pullprefetch logic to after the pull command is completely
finished. Updated a test to make sure this is caught.
Also fixes a bug where we were using linkrevs to read a manifest rev entry. We
should be using the manifest rev instead.
Test Plan: Added a test. Ran it.
Reviewers: sid0, pyd, davidsp
Differential Revision: https://phabricator.fb.com/D1483345
Summary:
Due to a change in upstream mercurial, hg log with patterns was no longer
working. This fixes it by forcing hg log to take the slow path when using
patterns.
It also updates the warning messages to work when running hg log <file> from
within a subdirectory.
Test Plan: Ran the new tests
Reviewers: sid0
Differential Revision: https://phabricator.fb.com/D1450193
Summary:
Adds a remotefilelog.pullprefetch config options that accepts a revset. Whenever
a pull is run, the revs matched by that revset will be prefetched. The most
common value for this will be '(bookmark() + heads(all())) & public()', since it will download
almost everything necessary to work offline.
Test Plan: Added a test. Ran it.
Reviewers: davidsp, pyd, sid0
Reviewed By: sid0
Differential Revision: https://phabricator.fb.com/D1419420
test-local.t was flakey because the state of the dirstate is non-deterministic
if files are written out within the same second. This rebuilds the dirstate
entirely in a few cases, which removes the flakeyness.
Summary:
Recent changes to upstream Mercurial have moved localrepo.getbundle and
localrepo.addchangegroupfiles to changegroup.py. remotefilelog wraps these
functions, and thus needs to be updated.
Applyupdate also had a function signature change, which is fixed here.
Minor fix to a test as well, which had a hard coded time instead of a glob.
Test Plan: ./run-tests.py --with-hg=/data/users/durham/hg/hg
Reviewers: sid0, davidsp, pyd, dschleimer
Differential Revision: https://phabricator.fb.com/D1260737
Preivously shallow clones only work using the streaming clone protocol. With
this change they work for the standard getbundle protocol as well. This is what
the majority of Mercurial users use, so we need to support that.
Summary:
hg bundle was producing shallow bundles. This change makes it produce full
sized bundles so they can be used in other repos.
Test Plan: Added a test
Reviewers: sid0
Reviewed By: sid0
CC: keegancsmith
Differential Revision: https://phabricator.fb.com/D1167462
Summary:
Previously requesting remotefilelog file blobs from the server required write
access in order to write the blob to the cache. This changes it to not abort
entirely if the user doesn't have write access.
Test Plan:
cd tests
./run-tests.py --with-hg=/data/users/durham/hg/hg test-permissions.t
Also ran the test without the fix and verified it fails.
Reviewers: sid0, davidsp, pyd, dschleimer
Reviewed By: dschleimer
Differential Revision: https://phabricator.fb.com/D1145976
Task ID: 3601184
The remotecmd config is no longer necessary since we now use a dummy ssh. It
was also hardcoded for a specific filesystem structure, which didn't work on
other machines.
Summary:
Adds a 'hg prefetch' command to remotefilelog for prepopulating the
local cache. Supports specifying revsets and file patterns to limit what is
downloaded.
Test Plan: ./run-tests.py test-prefetch.t --with-hg=/data/users/durham/hg/hg
Reviewers: dschleimer, sid0, davidsp, pyd, mpm
CC: kunalb, minyoung
Differential Revision: https://phabricator.fb.com/D1129942
Enables specifying a name for a repo that is used in the cache key.
This allows multiple repos on a machine to share a cache without the
risk of keys overlapping.
