Commit Graph

32 Commits

Author SHA1 Message Date
Durham Goode
201d5f2149 remotefilelog: fix getchangegroup signature
Summary:
The upstream getchangegroup function changed back in August and our attempt at
fixing it here did not correclty handle the case where remotefilelog is loaded
but not enabled for this repository.

This patches fixes the signature, and makes our wrapping more resitant to future
signature changes.

Test Plan: Added a test.

Reviewers: #mercurial, jsgf

Reviewed By: jsgf

Subscribers: jsgf, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4099519

Signature: t1:4099519:1477698801:adcb406f1908c4f7e508e5f99e126f383c6f2574
2016-10-31 13:40:33 -07:00
Durham Goode
f43ba75915 remotefilelog: fix pyflakes and module import errors
Summary:
This fixes all the pyflaks and module errors for the main remotefilelog
code base.

Test Plan: ./run-tests.py test-check* test-remotefilelog*

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4055537

Signature: t1:4055537:1477049663:ee904d311d17d3659e055e2c109c68c9023cfd1f
2016-10-21 11:02:09 -07:00
Ryan McElroy
5e7e9048ae remotefilelog: fix for upstream api change
Summary:
In upstream mercurial, c472ca028b32 changed changegroup.getchangegroup() to not
take a 'common' argument anymore.

Test Plan: run tests, no more stack traces

Reviewers: #mercurial, ttung, durham, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D3819304

Signature: t1:3819304:1473091041:587ecb88dfcdd806d372825ec4d7512eb7fdb94e
2016-09-05 12:02:28 -07:00
Martin von Zweigbergk
0df928d828 shallowbundle: specifically compare instance to remotefilelog.remotefilelog
In two place, we were checking if a revlog was an instance of
revlog.revlog and, I think, treating it as a
remotefilelog.remotefilelog otherwise. I noticed this when I created
another non-revlog.revlog revlog in narrowhg and remotefilelog thought
it was a remotefilelog.remotefilelog. Let's specifically check if it's
a remotefilelog.remotefilelog instead.
2016-07-15 23:53:09 -07:00
Durham Goode
8f4d83edeb shallowbundle: fix broken fallback orig call
This was caught by tests running in an unusual configuration
2016-04-28 17:34:08 -07:00
Durham Goode
c3c047f0b7 Move sortnodes into shallowutil
Summary:
This is a generic topological sort and will be useful in the upcoming repacking
code.

Test Plan: Ran the tests

Reviewers: #sourcecontrol, ttung

Reviewed By: ttung

Differential Revision: https://phabricator.fb.com/D3204124

Signature: t1:3204124:1461260520:e1cb5c9d496f11e5f44e0cdbc5ba851b1573d2e1
2016-04-26 15:10:38 -07:00
Durham Goode
3817826242 checkcode: fix remotefilelogserver and shallowbundle
Summary: Fix failures found by check-code.

Test Plan: Ran the tests

Reviewers: #sourcecontrol, ttung

Reviewed By: ttung

Differential Revision: https://phabricator.fb.com/D3221373

Signature: t1:3221373:1461648284:23203c17f4a87e33ff4e9be17a8b99bddbcdff05
2016-04-26 13:00:31 -07:00
Durham Goode
ece19111e0 ioutil: rename ioutil to shallowutil
The old name was not very descriptive. There's already a shallowutil, so let's
just use that.
2016-04-04 16:26:12 -07:00
Durham Goode
492b9af06e ioutil: move helper functions to ioutil
Future patches will refactor the storage into more abstract APIs. Let's move
these utility functions out to be on their own.
2016-04-04 16:26:12 -07:00
Mateusz Kwapich
cc54a98956 addchangegroup: adjust for new upstream API
Summary:
addchangegropfiles doesn't take the pr function as a parameter anymore.
The upstream change https://selenic.com/hg/rev/982e3ef7f5bf

Test Plan: tests are passing now on the release branch

Reviewers: #sourcecontrol, ttung, durham

Reviewed By: durham

Differential Revision: https://phabricator.fb.com/D3107217

Signature: t1:3107217:1459211189:4ece7531aff6043fc3acbfe43e2f471781c25c9d
2016-03-30 14:17:49 -07:00
Laurent Charignon
af9917b578 remotefilelog: fix compat with core on builddeltaheader 2015-12-30 13:33:47 -08:00
Martin von Zweigbergk
1c64f784ed make changegroup.addchangegroupfiles() overriding more flexible
The method gained a parameter in hg revision 43d86cd9dae2
(changegroup: note during bundle apply if the repo was empty,
2015-12-02).
2015-12-10 17:25:14 -08:00
Durham Goode
bb8c595d67 Update to work with latest Mercurial
Upstream Mercurial has made a lot of changes around streaming clones, so we need
to update remotefilelog to handle these new changes.
2015-10-13 14:17:02 -07:00
Durham Goode
5bb4351364 prefetch: add prefetching to bundle receiving
We were not prefetching the potential dependent files for the filelog revisions
we received over the wire. This resulted in a lot of non-batched downloads,
which was super slow. This fixes it by batch downloading the parents and delta
parents of the incoming filelog revisions and adds a test.
2015-07-21 18:32:33 -07:00
Laurent Charignon
5652bd276a Match with with latest version of core to pass the test
Summary:
Match with with latest version of core to pass the test.
There were a couple of changes in core that broke the extension, I matched
those changes to make the test pass.

Test Plan: The tests are all passing

Reviewers: durham

Differential Revision: https://phabricator.fb.com/D2053958
2015-05-07 12:50:51 -07:00
Siddharth Agarwal
8b622893dc [shallowbundle] don't drop units and reorder on the floor
Summary: We were forgetting to pass these arguments on to the child function.

Test Plan: Visual inspection.

Reviewers: durham, davidsp, rmcelroy, akushner, pyd, daviser, mitrandir, ericsumner

Reviewed By: ericsumner

Differential Revision: https://phabricator.fb.com/D1773782

Signature: t1:1773782:1420765574:d73be08ab25265e4769d8bf70671f2ea1c13f8dd
2015-01-08 17:02:37 -08:00
Siddharth Agarwal
d731468f70 [bundle2] insert ourselves into the cg1packer class hierarchy and fix up the packermap
Summary: Last bits needed to get remotefilelog over bundle2 working. Includes tests.

Test Plan: Ran tests, including with `--extra-config-opt experimental.bundle2-exp=True`

Reviewers: davidsp, akushner, pyd, rmcelroy, daviser, durham

Reviewed By: durham

Differential Revision: https://phabricator.fb.com/D1671738

Tasks: 5568731

Signature: t1:1671738:1415676482:b9e7a1f308919526b0c41fee54d89da876518ec7
2014-11-07 18:35:52 -08:00
Siddharth Agarwal
ca3f7a704e [bundle2] rename shallowbundle to shallowcg1packer
Summary: Preparation for bundle2 support

Test Plan: Ran tests

Reviewers: pyd, akushner, davidsp, durham

Reviewed By: durham

Differential Revision: https://phabricator.fb.com/D1668145

Tasks: 5568731

Signature: t1:1668145:1415643197:05ea239c2eb713f82bed6ad67bcd02fad7073a1f
2014-11-07 15:39:20 -08:00
Durham Goode
dc5a3bf415 Allow pulling from shallow bundlerepos
Bundlerepos work by providing a fake revlog layer above an existing revlog.
Since remotefilelog doesn't use revlogs for filelogs, bundlerepo's did not work.
This commit fixes it such that you can now hg pull from a bundle, as long as
that bundle is shallow (i.e. contains no file contents). This will work for the
common use case of trying to recover data from .hg/strip-backups.

For reference, shallow bundles don't contain any file data because we never
delete any file data from .hg/store/data when using remotefilelog.  Even after
the commits have been stripped.
2014-10-23 00:01:21 -07:00
Durham Goode
580f3eaeb3 Update to match Mercurial version b8c8cacd4482
Summary:
Changegroups have been refactored upstream and we need to update our
remotefilelog monkey patching accordingly.

Also fix an issue with the tests where 'function foo()' was not considered valid
on certain systems.

Test Plan: Ran the tests

Reviewers: pyd, sid0, davidsp

Differential Revision: https://phabricator.fb.com/D1551019
2014-09-11 14:39:14 -07:00
Siddharth Agarwal
ef8674624a Fix shallowbundle.getbundle for local non-remotefilelog repositories
Summary: Pulling from a local non-remotefilelog repo to a remotefilelog repo was broken. This fixes it.

Test Plan: `hg pull` from a local non-remotefilelog repo to a remotefilelog repo.

Reviewers: durham

Reviewed By: durham

Differential Revision: https://phabricator.fb.com/D1341059
2014-05-20 20:56:19 -07:00
Durham Goode
c5b2f574a0 Fix changegroup wrapping with new upstream Mercurial
Summary:
Recent changes to upstream Mercurial have moved localrepo.getbundle and
localrepo.addchangegroupfiles to changegroup.py.  remotefilelog wraps these
functions, and thus needs to be updated.

Applyupdate also had a function signature change, which is fixed here.

Minor fix to a test as well, which had a hard coded time instead of a glob.

Test Plan: ./run-tests.py --with-hg=/data/users/durham/hg/hg

Reviewers: sid0, davidsp, pyd, dschleimer

Differential Revision: https://phabricator.fb.com/D1260737
2014-04-04 15:55:06 -07:00
Durham Goode
bdea38dd56 Move fileservice to be per repo instead of global
Previously the file service client was a global object that all repos could
share. This was a bit hacky and is no longer needed. Now the file service
client exists per repo instance.

This is part of a series of changes to abstract the local caching and remote
file service in such a way that we can plug and play implementations.
2014-02-11 14:41:56 -08:00
Durham Goode
fc3a887712 hg bundle produces full sized bundles
Summary:
hg bundle was producing shallow bundles. This change makes it produce full
sized bundles so they can be used in other repos.

Test Plan: Added a test

Reviewers: sid0

Reviewed By: sid0

CC: keegancsmith

Differential Revision: https://phabricator.fb.com/D1167462
2014-02-10 16:13:41 -08:00
Durham Goode
d9d4477013 Remove global variable for tracking shallow remotes
Previously we used a global variable to track if the incoming connection was
from a shallow remote (based on if the network command was a *_shallow command).
This is hacky and overall a bad idea. The new implementation stores the shallow
flag as a bundlecapability passed to the getbundle command.

A side effect of this is remotefilelog won't work with versions of mercurial
that don't use the getbundle command.
2013-11-25 14:22:56 -08:00
Durham Goode
1275d15990 Add include and exclude configuration settings
The remotefilelog extension currently doesn't work with tags. Adding include and
exclude patterns allows users to specify which files they want to treat as
shallow and which the want to download the entire history for. By excluding
.hgtags from being shallow, this enables tags to work in a mostly shallow repo.

This also enables largefile like scenarios where most files are full and only a
few large ones are kept remote.
2013-09-26 10:46:06 -07:00
Durham Goode
efdfcc1502 Send all available data during a pull 2013-09-19 16:22:14 -07:00
Durham Goode
6781d80d25 Fix local pulls to send file data 2013-09-09 11:44:08 -07:00
Durham Goode
3c879ed1a8 Enable efficient pulling between shallow repos 2013-08-28 18:51:01 -07:00
Durham Goode
3e6b7810df Override bundle10.generatefiles instead of prune 2013-06-25 13:26:24 -07:00
Durham Goode
84b481de56 Add option for server cache location.
Change _callstream wrapper to only run on client.
2013-06-21 13:22:18 -07:00
Durham Goode
f16a3a4134 Rename to remotefilelog since shallowrepo is already taken 2013-06-21 10:14:29 -07:00