Summary: Last bits needed to get remotefilelog over bundle2 working. Includes tests.
Test Plan: Ran tests, including with `--extra-config-opt experimental.bundle2-exp=True`
Reviewers: davidsp, akushner, pyd, rmcelroy, daviser, durham
Reviewed By: durham
Differential Revision: https://phabricator.fb.com/D1671738
Tasks: 5568731
Signature: t1:1671738:1415676482:b9e7a1f308919526b0c41fee54d89da876518ec7
Summary:
- fixes for pulls after uncompressed clones
- add a glob where there should be one
Test Plan: Ran the tests with Mercurial @
Reviewers: davidsp, akushner, rmcelroy, pyd, daviser, durham
Reviewed By: durham
Differential Revision: https://phabricator.fb.com/D1668739
Signature: t1:1668739:1415643384:567a0841ef4367ee2f905f6fe5517d229d5eadcf
Certain filectx constructions used the rev number of the self._changeid. We
need to convert that to a node before using it. This was breaking blame. I've
now added a blame test too.
Bundlerepos work by providing a fake revlog layer above an existing revlog.
Since remotefilelog doesn't use revlogs for filelogs, bundlerepo's did not work.
This commit fixes it such that you can now hg pull from a bundle, as long as
that bundle is shallow (i.e. contains no file contents). This will work for the
common use case of trying to recover data from .hg/strip-backups.
For reference, shallow bundles don't contain any file data because we never
delete any file data from .hg/store/data when using remotefilelog. Even after
the commits have been stripped.
Upstream Mercurial has moved localrepo.pull into exchange.pull. This moves our
wrapping of that command out of shallowrepo and into __init__. Exchange is
becoming an increasingly important class, so we may want to think about moving
all exchange wrapper logic out to a separate module in remotefilelog.
repo.revs() no longer returns an object that can be indexed, so we can't use []
on it anymore. So let's use list() on it first.
The bookmark output from upstream Mercurial has also changed, so we need to
update the tests.
Summary:
When doing 'hg unshelve foo.txt' with Changeset Evolution enabled, uncommit will
first prune the commit, then try to read the filelog history to determine if any
renames need to be undone. Since the commit is now pruned, remotefilelog fails
to find any valid histories.
This fixes it two allow hidden histories if the filectx commit is hidden. It
also tweaks remotefilectx to produce commit-relative histories when possible,
which will result in more accurate histories.
Test Plan:
Ran hg uncommit in the evolve repo that had problems before. Verified
it now worked.
Reviewers: pyd, sid0
Differential Revision: https://phabricator.fb.com/D1587306
Summary:
Pull-prefetch would not download file versions from the server if the file
version already existed in the local cache or the local store data.
Unfortunately, if someone landed their commit, then later stripped their local
version, the local store data file version might become invalid and no local
cache version would exist. Meaning things like 'commit' might fail when offline.
This changes prefetch to always fetch from the server when dealing with files it
knows are from revs on the server.
Test Plan:
Added a test that makes local commits that already exist on the
server, and verifies that a pull-prefetch fetches the server file version,
despite that same version existing locally.
Reviewers: sid0, pyd, davidsp
Subscribers: orip
Differential Revision: https://phabricator.fb.com/D1607260
Summary:
The ResponseError exception expect a second argument. Otherwise the code
handling it crashes.
Test Plan: The handling of the response error stop crashing.
Reviewers: durham
Differential Revision: https://phabricator.fb.com/D1581574
Summary:
If the orig function crash before the fileservice is installed, the finally
clause explode, shadowing the original error. We fixes thats.
Test Plan:
crash stopped being shadowed but crash in the finally clause.
Reviewers: durham
Differential Revision: https://phabricator.fb.com/D1581562
Summary: API change
Test Plan: @durham ran an amend.
Reviewers: durham
Reviewed By: durham
Subscribers: durham
Differential Revision: https://phabricator.fb.com/D1569510
Summary:
Upstream Mercurial changed the way merging works and added
revlog.commonancestorsheads. This changes remotefilelog to implement the same
API.
Previously we were able to use ancestors.genericancestors to do the graph
traversal. Upstream Mercurial has deleted that function though (since it is now
unused), so remotefilelog must now build a temporary rev graph in order to use
the ancestors.* apis.
Test Plan: Added a test. It failed without the fix, it passes with the fix.
Reviewers: sid0, davidsp, pyd
Differential Revision: https://phabricator.fb.com/D1566787
Summary: This was broken by recent changes.
Test Plan: Ran test suite.
Reviewers: durham
Reviewed By: durham
Differential Revision: https://phabricator.fb.com/D1558890
Tasks: 5170539
Summary:
With recent version of mercurial (>= 3.2, 4dfcf21a6aa7), revert uses status
information to determine the files that needs to be touched. It then offer a
simple handle for extensions that needs prefetch.
Test Plan:
Ran the tests. Certain tests depended on the old revert behavior (of
prefetching everything), so they required slight changes.
Reviewers: pyd, sid0, davidsp
Differential Revision: https://phabricator.fb.com/D1551059
Summary:
Changegroups have been refactored upstream and we need to update our
remotefilelog monkey patching accordingly.
Also fix an issue with the tests where 'function foo()' was not considered valid
on certain systems.
Test Plan: Ran the tests
Reviewers: pyd, sid0, davidsp
Differential Revision: https://phabricator.fb.com/D1551019
Summary:
Previously, if pullprefetch was set, we'd perform a prefetch of the
entire manifest of the specified revs (usually the public bookmarks). This
involved stat-ing all the relevant files in the cache to see if they already
existed, which added an extra 6 seconds or so to every pull.
Now we only prefetch the files that are different from our working copy. We
assume we already have all the files that are in our working copy. This reduces
the pullprefetch overhead significantly.
Test Plan:
Did a pull on my laptop. Verified it didn't hang for 6 seconds at the
prefetch stage. Also updated a test
Reviewers: davidsp, pyd, sid0
Reviewed By: sid0
Differential Revision: https://phabricator.fb.com/D1505841
Tasks: 4608894
Summary:
Previously, pullprefetch was executed during the repo.pull stage. This happens
before the bookmarks have been moved, so revsets like 'bookmark()' would
prefetch the wrong commits.
This change moves the pullprefetch logic to after the pull command is completely
finished. Updated a test to make sure this is caught.
Also fixes a bug where we were using linkrevs to read a manifest rev entry. We
should be using the manifest rev instead.
Test Plan: Added a test. Ran it.
Reviewers: sid0, pyd, davidsp
Differential Revision: https://phabricator.fb.com/D1483345
Summary: These commands (well, not the debug one) were visible in the shortlist that showed up when you type `hg`. They're not basic commands.
Test Plan: Ran `hg` with the extension enabled, didn't see those commands.
Reviewers: durham
Reviewed By: durham
Differential Revision: https://phabricator.fb.com/D1454931
Summary:
Due to a change in upstream mercurial, hg log with patterns was no longer
working. This fixes it by forcing hg log to take the slow path when using
patterns.
It also updates the warning messages to work when running hg log <file> from
within a subdirectory.
Test Plan: Ran the new tests
Reviewers: sid0
Differential Revision: https://phabricator.fb.com/D1450193
Summary:
Adds a remotefilelog.pullprefetch config options that accepts a revset. Whenever
a pull is run, the revs matched by that revset will be prefetched. The most
common value for this will be '(bookmark() + heads(all())) & public()', since it will download
almost everything necessary to work offline.
Test Plan: Added a test. Ran it.
Reviewers: davidsp, pyd, sid0
Reviewed By: sid0
Differential Revision: https://phabricator.fb.com/D1419420
Summary:
Expands environment variables in the cacheprocess and cachepath config options,
so users can specify something like remotefilelog.cachepath=$HOME/.hgcache
Test Plan:
Set my cachepath to $HOME/.hgcache on my laptop and manually
performed a shallow clone. Verified data was put in ~/.hgcache
Reviewers: sid0
Differential Revision: https://phabricator.fb.com/D1342174
test-local.t was flakey because the state of the dirstate is non-deterministic
if files are written out within the same second. This rebuilds the dirstate
entirely in a few cases, which removes the flakeyness.
Summary: Pulling from a local non-remotefilelog repo to a remotefilelog repo was broken. This fixes it.
Test Plan: `hg pull` from a local non-remotefilelog repo to a remotefilelog repo.
Reviewers: durham
Reviewed By: durham
Differential Revision: https://phabricator.fb.com/D1341059
Summary:
Recent changes to upstream Mercurial have moved localrepo.getbundle and
localrepo.addchangegroupfiles to changegroup.py. remotefilelog wraps these
functions, and thus needs to be updated.
Applyupdate also had a function signature change, which is fixed here.
Minor fix to a test as well, which had a hard coded time instead of a glob.
Test Plan: ./run-tests.py --with-hg=/data/users/durham/hg/hg
Reviewers: sid0, davidsp, pyd, dschleimer
Differential Revision: https://phabricator.fb.com/D1260737
Preivously shallow clones only work using the streaming clone protocol. With
this change they work for the standard getbundle protocol as well. This is what
the majority of Mercurial users use, so we need to support that.
The current local cache is just files on disk, and this implementation detail
was spread across the extension. This change refactors it to hide the
implementation inside a class so that we can replace it with other
implementations (such as a sqlite local cache) later.
Previously the file service client was a global object that all repos could
share. This was a bit hacky and is no longer needed. Now the file service
client exists per repo instance.
This is part of a series of changes to abstract the local caching and remote
file service in such a way that we can plug and play implementations.
If the memcache process exited early, remotefilelog was throwing an exception
instead of falling back to the server. This change makes it fall back to the
server, and also print a warning that the cache connection closed early.
Summary:
hg bundle was producing shallow bundles. This change makes it produce full
sized bundles so they can be used in other repos.
Test Plan: Added a test
Reviewers: sid0
Reviewed By: sid0
CC: keegancsmith
Differential Revision: https://phabricator.fb.com/D1167462
Summary:
Previously requesting remotefilelog file blobs from the server required write
access in order to write the blob to the cache. This changes it to not abort
entirely if the user doesn't have write access.
Test Plan:
cd tests
./run-tests.py --with-hg=/data/users/durham/hg/hg test-permissions.t
Also ran the test without the fix and verified it fails.
Reviewers: sid0, davidsp, pyd, dschleimer
Reviewed By: dschleimer
Differential Revision: https://phabricator.fb.com/D1145976
Task ID: 3601184
The remotecmd config is no longer necessary since we now use a dummy ssh. It
was also hardcoded for a specific filesystem structure, which didn't work on
other machines.
Summary:
Adds a 'hg prefetch' command to remotefilelog for prepopulating the
local cache. Supports specifying revsets and file patterns to limit what is
downloaded.
Test Plan: ./run-tests.py test-prefetch.t --with-hg=/data/users/durham/hg/hg
Reviewers: dschleimer, sid0, davidsp, pyd, mpm
CC: kunalb, minyoung
Differential Revision: https://phabricator.fb.com/D1129942
The alternate lookup code was mistakening looking for only the last digit
instead of looking at the entire prefix. This meant files with more than 10
alternates would start failing to find histories, which breaks rebase.
When falling back to the master server for cache misses, we only kept two
requests in flight at any time. Over high latency connections (like across
oceans) this resulted in very slow downloads.
This change increases the request size to 10,000 keys at once. This will keep
the size of the request lower than the tcp buffer size, while allowing us to
maximize our throughput.
Previously we sent the entire list of files to the fallback repo in a single ssh
write/flush. If the size of this write exceeded the tcp buffer on the receiving
end, the call would hang until the buffer had room. The problem is that the
receiving end (the server) is hung trying to send data back to the
client. Therefore it deadlocked.
The fix is to send and receive requests one at a time. We always have the next
request in flight while receiving so we shouldn't be waiting on requests too
often.
Enables specifying a name for a repo that is used in the cache key.
This allows multiple repos on a machine to share a cache without the
risk of keys overlapping.
Previously we used a global variable to track if the incoming connection was
from a shallow remote (based on if the network command was a *_shallow command).
This is hacky and overall a bad idea. The new implementation stores the shallow
flag as a bundlecapability passed to the getbundle command.
A side effect of this is remotefilelog won't work with versions of mercurial
that don't use the getbundle command.
The previous algorithm thought that if the system cache had the file rev, it was
guaranteed to be valid. This isn't true in the case of a machine in which
multiple people share the cache (one person may have pulled a rev but the other
hasn't).
The new algorithm is more explicit. It checks:
- system cache
- local cache
- local cache fallbacks
- remote cache
- master server
