Previously test-treemanifest-treeonly.t was hanging. Now it doesn't hang (but
still fails).
It obviously doesn't fix the real issue, but at least it doesn't make test
hang.
Test Plan:
Run test-treemanifest-treeonly.t
Differential Revision: https://phab.mercurial-scm.org/D1205
When calling prefetch in remotefilelog, also prefetch lfs
files.
We are using the same hook mechanism that remotefilelog is already using
for LFS by having remotefilelog call into LFS.
Test Plan:
run tests on test-lfs-remotefilelog-prefetch.t
Differential Revision: https://phab.mercurial-scm.org/D732
Upstream has changed local bundles to use the highest available changegroup
version, which means we're using changegroup3 for strips now. So we need to
handle that in shallowbundle.
Summary:
This functionality allows to add delays between background prefetches after
operations that change the working copy parent. By default background
prefetches will be run no often than every 2 minutes, but this is configurable.
This allows to reduce the load on CPU(because prefetch is followed by repack).
Test Plan:
Tested manually on fbsource repo by running hg co bookmark several
times and ensuring that only 1 background prefetch is running if 2 minute
timeout has not expired.
Reviewers: durham, simonfar, ryanmce, #fbhgext
Reviewed By: simonfar, #fbhgext
Differential Revision: https://phab.mercurial-scm.org/D712
Summary: `test-check-code-hg.t` complains about long lines here.
Test Plan: - rt, make sure it does not complain anymore
Reviewers: durham, #fbhgext
Differential Revision: https://phab.mercurial-scm.org/D697
Summary: Prevents downloading any manifest revlogs when the client is in treeonly mode.
Reviewers: #fbhgext, simonfar
Reviewed By: #fbhgext, simonfar
Differential Revision: https://phab.mercurial-scm.org/D575
Summary:
Keepset computation algorithm added tuples to the keepset that are the results
of diffs between two manifests. Now it has been fixed and only new node from
the tuple is added to the keepset.
Test Plan:
* Run hg gc on www, fbsource
* Ensure the unit tests pass
Reviewers: simonfar, durham, #fbhgext, ryanmce
Reviewed By: #fbhgext, ryanmce
Subscribers: ryanmce
Differential Revision: https://phab.mercurial-scm.org/D519
Summary:
Previously incremental repack would only take the top 3 packs in a generation of
packs. For instance, if there were 100 packs between 1KB and 1MB, it would only
repack the first 3, instead of repacking a whole bunch because it's cheap. This
patch fixes it to grab as many packs in a generation as it can fit under the
limit (default of 100MB of pack data).
This will fix an issue where we're seeing users with 100's of tiny pack files
that aren't getting repacked fast enough.
Test Plan: Added a test
Reviewers: #fbhgext, quark
Reviewed By: #fbhgext, quark
Subscribers: quark
Differential Revision: https://phab.mercurial-scm.org/D500
D319 renamed `peer.pipe[ioe]` to `peer._pipe[ioe]`.
D320 removed `peer.batch`.
D331 renamed `subprocess` to `_subprocess`.
D336 renamed `_capabilities` to `capabilities`.
Let's be compatible with those changes.
Differential Revision: https://phab.mercurial-scm.org/D425
Summary:
New keepset computation algorithm yields 50% time improvement on a small set of
commits and about 90% time improvement on a large set of commits. Currently it
takes about 1m10s to compute a keepset of 10 commits with treemanifest
usecunionstore flag unset and 3 seconds with cunionstore config set, provided
all trees are prefetched.
Test Plan: * Ensure that unit tests pass, in particular *-gc.t and *-gcrepack.t
Reviewers: ryanmce, simonfar, durham, #fbhgext
Reviewed By: ryanmce, #fbhgext
Differential Revision: https://phab.mercurial-scm.org/D442
Summary:
Added config option 'prefetchdays' which indicates that commits older than
'prefetchdays' days should not be prefetched. This option is necessary to avoid
prefetch of huge amount of data. The default value is set to 14 days.
Test Plan: Ensure that unit tests pass
Reviewers: ryanmce, simonfar, durham, #fbhgext, simpkins
Reviewed By: #fbhgext, simpkins
Subscribers: simpkins
Differential Revision: https://phab.mercurial-scm.org/D420
Summary:
repo.revs cannot resolve aliased revisions such as 'top' or 'bottom', therefore
it is substituted by scmutil.revrange for any user provided inputs.
Test Plan:
* Test on www repo by running hg pull with remotefilelog.pullprefetch='bottom'(or any other aliased rev)
* Ensure that unit tests pass
Reviewers: simonfar, rmcelroy, durham
Reviewed By: durham
Subscribers: medson, mjpieters, #mercurial
Differential Revision: https://phabricator.intern.facebook.com/D5602536
Signature: t1:5602536:1502386617:b632f778da4ddef54d7283b60c7c1a24c6b38028
Summary: Now on hg gc incremental repack is run if configured
Test Plan: Added a test case
Reviewers: simonfar, durham
Reviewed By: durham
Subscribers: medson, mjpieters, #mercurial
Differential Revision: https://phabricator.intern.facebook.com/D5583902
Tasks: 19727557
Signature: t1:5583902:1502221690:dce09112207e9e4595458d61a7b73f1210d3b012
Summary:
Now hg gc keepset includes pullprefetch and bgprefetchrevs revsets to avoid
garbage collection of prefetched in background data.
Test Plan: Added test case, also tested in *-gcrepack.t
Reviewers: durham, simonfar
Reviewed By: simonfar
Subscribers: medson, mjpieters, #mercurial
Differential Revision: https://phabricator.intern.facebook.com/D5583184
Tasks: 19727557
Signature: t1:5583184:1502208410:c247d802cf27c0143b6d7e2d95029bad5f344499
Summary:
Implemented initial version of a garbage collector for repack. Garbage
collection is only performed if gcrepack flag is set in config. Currently
data that is old and not in keepset is garabage collected. The age of data at
which point it will be garbage collected can be specified in config as well.
Test Plan: Added a test case
Reviewers: durham
Reviewed By: durham
Subscribers: rmcelroy, medson, mjpieters, #mercurial
Differential Revision: https://phabricator.intern.facebook.com/D5548194
Tasks: 19727521
Signature: t1:5548194:1501882489:cc775ec95aa6fafc462a46f99d790f231139ce27
Summary:
If bgprefetchrevs is specified in config, background prefetch will be executed on
update and commit commands as well as on other commands which use update or
commit. Background prefetch will fetch revisions specified in config and if the
repack flag is set will also execute a repack.
Test Plan: Added test cases
Reviewers: simonfar, durham
Reviewed By: durham
Subscribers: medson, mjpieters, #mercurial
Differential Revision: https://phabricator.intern.facebook.com/D5454939
Tasks: 19727407
Signature: t1:5454939:1500551182:e9cce84f6dc98182b5cb30faeb811fd7fa5e22b0
Summary:
Previously prefetch on pull ran in foreground. Now it can run in background
if specified. Optionally background prefetch can be followed by background
repack.
Test Plan: Added a test case
Reviewers: simonfar, durham
Reviewed By: simonfar
Subscribers: medson, mjpieters, #mercurial
Differential Revision: https://phabricator.intern.facebook.com/D5406626
Tasks: 19727343
Signature: t1:5406626:1500059334:207b4100cca536cbe33f6c6dfad596d03a6fa14f
Summary:
When committing a node which already exists, the linkrev provided to filelog
is actually wrong (len(changelog)). The upstream Mercurial will not update
existing file revisions but remotefilelog will.
The fact that linkrev being wrong triggers the ProgrammingError added by
D5061330. This diff detects that case and avoid writing bad linkrevs or
raise ProgrammingError.
Test Plan: Added a test.
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: medson, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5278060
Signature: t1:5278060:1497910883:b8c3b6281ad9c0a516b942ba8ca41c51801b2d6b
Summary: Upstream has deprecated cmdutil.commands() in favor of registrar.commands()
Test Plan: Ran the tests
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5106486
Signature: t1:5106486:1495485074:0e20f00622cc651e8c9dda837f84dd84cc51099e
Summary:
Previously our data packs would have incredibly long delta chains, where to read
the last entry in the file you had to read every previous delta all the way to
the beginning. This was very expensive for chains of 2+ million deltas. Let's
add a config option to limit how long the deltas get, and for now we'll default
it to 1000.
Test Plan: Added a test
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5095603
Signature: t1:5095603:1495213707:737d63129cf459ad6927a1f4deb0dda5a8ce0a7f
Summary:
remotefilelog needs to wait for changelog creation to get the commit hash so
all filelog appending operations are pending if linkrev is an integer.
That is better done in the `addrawrevision` layer since it will catch more code
paths. Besides, that means remotefilelog won't need to hash a same revision
twice.
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5061330
Signature: t1:5061330:1494871717:a7224ebdc0f221fbaabbd2a58de975caec0e4b05
Summary:
The test covers common workflows like clone, commit, push, update, pull. It
exercises the remotefilelog plain store and Python datapack store to make sure
they won't lose the revlog flag. The test also tries to verify rename works
correctly.
Since the lfs extension may be eventually upstreamed, it seems a good idea to
make remotefilelog call `lfs.wrapfilelog` so lfs is free from remotefilelog
code.
Test Plan: Added a test
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4904281
Signature: t1:4904281:1492560308:5fd9f214ada6de795735ea7d737d30c1bf39812a
Summary:
`mercurial.revset` was recently split into `revset + revsetlang + smartset`.
Update our code accordingly.
D4604848 has fixed the `revsetlang` part. This patch fixes the remaining
`smartset` part.
Also fixes some test failures introduced by D4547080.
Test Plan: `arc unit`
Reviewers: #mercurial
Subscribers: jeroenv, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4595417
Summary:
The remotefilelog extension previously unconditionally added a sparsematch()
method to repository objects, which just called super.sparsematch() if this is
a sparse repository, and returned None if this isn't a sparse repository.
However, defining a sparsematch() method that returns None confuses the sparse
extension. The sparse extension expects that all repositories that define
sparesmatch() are actually sparse repositories, and never expects this method
to return null.
This updates the remotefilelog code to call its method maybesparsematch()
instead.
Test Plan:
Confirmed existing remotefilelog unit tests pass, and that the sparse extension
no longer crashes when it is used with non-sparse repositories. (This happens
when using the share extension, if the current working directory is non-sparse
but the sharedpath repository configuration loads the sparse extension.)
Reviewers: tja, durham
Reviewed By: durham
Subscribers: net-systems-diffs@, yogeshwer, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4466108
Signature: t1:4466108:1485451483:287a519151e35bdb99f5be0d9287b4698386183e
Summary:
Using `testedwith = 'internal'` is not a good habit [1]. Having it
auto-updated in batch would also introduce a lot of churn. This diff makes
them "ships-with-fb-hgext". If we do want to fill the ideal "testedwith"
information, we could put it in a centric place, like a "fbtestedwith"
extension rewriting those "ships-with-fb-hgext" on the fly.
Maybe having in-repo tags for tested Mercurial releases is also a good idea.
[1]: www.mercurial-scm.org/repo/hg/rev/2af1014c2534
Test Plan: `arc lint`
Reviewers: #sourcecontrol, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4244689
Signature: t1:4244689:1480440027:3dc18d017b48beba1176fbfd120351889259eb4b
Summary:
The debugindex and debugindexdot commands have moved and are not registered
unless you import the new mercurial.debugcommands module.
Test Plan:
Run
hg --config=extensions.remotefilelog=fb-hgext/remotefilelog help
and observe that you get help info, rather that the error
hg: unknown command 'debugindex'
then run the fb-hgext test suite.
Reviewers: rmcelroy, quark, simonfar
Reviewed By: simonfar
Subscribers: mjpieters, #mercurial
Differential Revision: https://phabricator.intern.facebook.com/D4244047
Signature: t1:4244047:1480427216:dcaa1ca441ea189bdf68f1f619b4078d8c1d09dc
Summary:
We've gotten reports of hg gc failing on some service machines because
`peer._repo.name` complains that repo has no attribute 'name'. I'm not sure how
this could happen, but it makes sense to make the hg gc loop more robust to the
possibility that the repos in the 'repos' file have changed their configuration
since they were added to the file.
Test Plan: Ran the tests
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4072719
Signature: t1:4072719:1477385020:24d532b9442292ce8234cc91bc7de503d3b0c88f
Summary:
Upstream changed the signature of computenonoverlap. Let's change our wrapping
of it to be more robust to signature changes.
Test Plan: ./run-tests.py test-remotefilelog* test-check*
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4062705
Tasks: 14037455
Signature: t1:4062705:1477096049:5a011a7a5edf9bb01475694777c738cdb02453f5
Summary:
This fixes all the pyflaks and module errors for the main remotefilelog
code base.
Test Plan: ./run-tests.py test-check* test-remotefilelog*
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4055537
Signature: t1:4055537:1477049663:ee904d311d17d3659e055e2c109c68c9023cfd1f
Summary:
764cd9916c94 recently introduced code that was unconditionally checking the
repo.includepattern and repo.excludepattern attributes on a local repository
without first checking if this is a shallow repository. These attributes only
exist on shallow repositories, causing "hg pull" to crash on non-shallow
repositories. This crash wouldn't happen in simple circumstances, since the
remotefilelog extension only gets fully set up once a shallow repository object
has been created, however when using chg you can end up with scenarios where a
non-shallow repository is used in the same hg process after a shallow one.
This refactors the code to now store the local repository object on the remote
peer rather than trying to store the individual shallow, includepattern, and
excludepattern attributes.
Overall this code does still feel a bit janky to me -- the rest of the peer API
is independent of the local repository, but the _callstream() wrapper cares
about the local repository being referenced. It seems like we should ideally
redesign the APIs so that _callstream() receives the local repository data as
an argument (or we should make the peer <--> local repository assocation more
formal and explicit if think it's better to force an association here).
Test Plan: Added a new test which triggered the crash, but passes with these changes.
Reviewers: ttung, mitrandir, durham
Reviewed By: durham
Subscribers: net-systems-diffs@, yogeshwer
Differential Revision: https://phabricator.intern.facebook.com/D3756493
Tasks: 12823586
Signature: t1:3756493:1471971600:9666e9c31bf59070c3ace0821d47d322671eb5b1
Previously we were kicking off background repacks even for non remotefilelog
repos. Moving the repack to be inside the remotefilelog requirement check will
prevent this.
Calling wrapfunction on the remotefilepeer(sshpeer) object in exchangepull
function introduces a reference cycle. Hence, this object will not be deleted
until the process dies. This is not a big issue for processes having a short
lifetime(e.g. lauched by command line.)
However, for persistent processes (e.g. TortoiseHg), this can lead to multiple
lingering ssh connections to the server(actually one by pull operation).
The fix is to not wrap the remotefilepeer._callstream. This method is defined
right into the remotefilepeer object. The required repo data is made available
in the remotefilepeer object by monkeypatching this object in the exchangepull
function.
In some situations the remotefilelog setup logic could be called, which will
wrap certain functions, and then later a call will happen to a repo that wasn't
remotefilelog which will run some remotefilelog code because of the wrapping.
Normally we take care of this by checking for the remotefilelog requirement. We
missed it in this one spot though.
Before this patch, debugremotefilelog and verifyremotefilelog would
crash if not given a path. Also, many commands would accept arguments
they then ignored.
Summary:
Previously a bunch of different places accessed the cachepath through ui.config
directly. This is a problem because we need to resolve any environment variables
in the path, and some spots didn't do this. So let's unify all accesses through
a helper function that takes care of the environment variables.
Test Plan: Added a test
Reviewers: mitrandir, lcharignon, #sourcecontrol, ttung, simonfar
Reviewed By: simonfar
Subscribers: simonfar
Differential Revision: https://phabricator.intern.facebook.com/D3385583
Signature: t1:3385583:1464971813:5b9ee5ed3d6ff9f1a78cb9e0269e433844758c9d
Previously, background repacks would only repack pack files, which meant there
was no automated way to repack loose remotefilelog files without manually
running 'hg repack'. This allows incremental repacks to also pack the loose
files.
It also changes the config knob for background repacks, so we can enable pack
file usage without the server having to support it just yet.
Summary:
This runs the incremental background repacking logic after hg pull.
As part of adding tests, I also added a 'hg debugwaitonrepack' function that
will wait until any pending repack is done before returning, so the tests can
wait on repacks without so many sleeps.
Test Plan: Adds a test
Reviewers: mitrandir, #mercurial, ttung, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy
Differential Revision: https://phabricator.intern.facebook.com/D3306526
Signature: t1:3306526:1463696933:9e27daf0c08076468e8f365a3c372fa7d4f56bde
Summary:
This adds a --incremental flag to the hg repack command. This flag causes repack
to look at the distribution of pack files in the repo and performs the most
minimal repack to keep the repo in good shape. Currently it's only implemented
for datapacks.
The new remotefilelog.datagenerations config contains a list of the sizes for
the different generations of pack files. For instance:
[remotefilelog]
datagenerations=1GB
100MB
1MB
Designates 4 generations - packs over 1GB, packs over 100MB, packs over 1MB, and
implicitly packs undex 1MB. The incremental algorithm will try to keep each
generation to less than 3 pack files (prioritizing the larger generations
first). When performing a repack it will grab at least 2 packs, and will grab
more if the total pack size is less than 100MB (since repacking at that level is
pretty cheap).
I have no idea if this is a good algorithm. We'll how to see and iterate.
Test Plan: Adds a test
Reviewers: mitrandir, #mercurial, ttung, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy
Differential Revision: https://phabricator.intern.facebook.com/D3306523
Signature: t1:3306523:1463697129:c87f4a397ef357b5ca4a80d01e9a6ca4d61f9d3d
Summary:
A future patch will be adding incremental repack, so let's move our repack logic
to the repack module so it's easier to refactor and extend.
Also adds a message for when a background repack kicks off (since we'll be
calling that from other places eventually).
Test Plan: Adds a test
Reviewers: mitrandir, #mercurial, ttung, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy
Differential Revision: https://phabricator.intern.facebook.com/D3306521
Signature: t1:3306521:1463602886:cece3d517f0672b829702866482c902812f9ae27
