Commit Graph

2190 Commits

Author SHA1 Message Date
Durham Goode
3ad41b8578 treemanifest: limit autocreatetrees to commits on master
Summary:
Creating a tree for a commit whose parent is not already a tree is expensive.
Let's optimize the autocreatetrees option (which converts manifests to trees
during pull) to only create trees if A) the parent is already a tree, or B) the
parent is master. This way we only pay the expensive part once. It also means
that as new branches fork off master, they will be trees too, since all commits
in the new branch will meet criteria (A).

Test Plan:
Ran hg pull in a large repo over a large pull with 40 different
branches and verified it only paused for the initial master commit.

Reviewers: #mercurial, simonfar

Reviewed By: simonfar

Subscribers: simonfar, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4129010

Signature: t1:4129010:1478260239:1698ada4e3c6a38ab77a94317e75daee4812276f
2016-11-16 13:51:48 -08:00
Adam Simpkins
4b57fdd6ae [fastmanifest] fix crash in hybridmanifest.__nonzero__()
Summary:
hybridmanifest.__nonzero__() was explicitly trying to call __nonzero__() on the
underlying manifest object.  This breaks when the underlying manifest is a
manifestdict.  manifestdict does not implement __nonzero__ (it instead has a
__len__ method that gets used instead when evaluating the manifestdict as a
boolean).

Because of this issue I was getting a crash in a local script when calling
repo.commitctx() with a memctx object.

Test Plan: Confirmed that calling repo.commitctx() with a memctx no longer crashes.

Reviewers: #sourcecontrol, durham, quark

Reviewed By: quark

Subscribers: net-systems-diffs@, yogeshwer, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4191858

Signature: t1:4191858:1479330743:7adb668c9529a8c8abaf9a8a1e64c8ee78b64d86
2016-11-16 13:16:31 -08:00
Durham Goode
b1109deca8 treemanifest: update to work with manifestlog
Summary:
Upstream has refactored the manifest class into several classes, so we need to
update treemanifest to work with the new structure. Notably, the factory add
function previously relied on the ability for the revlog class to create a new
manifestdict (via manifest.maniest.read()), since this isn't possible anymore,
we have to construct the hybridmanifest ourselves and provide an appropriate
loadflat function to get the flat manifest if necessary.

Test Plan: Ran the tests

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4180891

Signature: t1:4180891:1479285991:82bc546a1eb682d3cfd8b4724bda575410405d0f
2016-11-16 12:11:15 -08:00
Durham Goode
8e63595d58 catnotate: fix to work with manifestlog
Summary:
Upstream has refactored the manifest to be a manifestlog, so we need to update
our usage of it accordingly.

Test Plan: Ran the tests

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4180889

Signature: t1:4180889:1479228547:378f91b556493e05ff0222449b43cf498c301467
2016-11-16 12:11:13 -08:00
Durham Goode
7a97fe8dca pushrebase: fix to work with new manifestlog
Summary:
Upstream has refactored the manifest class, so we need to update pushrebase to
work with the new structure. In particular, pushrebase takes advantage of the
manifest cache, so we need to carefully reconstruct the appropriate cache on the
bundle repo. We can't just .copy() it now because the cache contains manifestctx
instead of manifestdict, and the manifestctx contains a reference to the source
repo.

Test Plan: Ran the tests

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4180885

Signature: t1:4180885:1479228506:a0ff114e158caab25da22f6116e5a4090f5515c0
2016-11-16 12:11:11 -08:00
Durham Goode
508d4b493b remotefilelog: update to work with manifestlog
Summary:
Upstream has refactored the manifest class, so we need to update remotefilelog
to work with the new structure.

Also fix a mismatch with the new adjustlinkrev signature.

Test Plan: Ran the tests

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4180879

Signature: t1:4180879:1479285946:d5ac954c495dc8b802d276ab5ac71626a1726612
2016-11-16 12:11:08 -08:00
Durham Goode
4ddfb704d3 fastmanifest: update to work with manifestlog
Summary:
Upstream Mercurial has refactored the manifest to get rid of the manifest class.
This patch updates fastmanifest to work with the new class structure. In
particular, it removes the hacky wrapping of 3 different manifest construction
functions, with a single wrapping of manifestlog.get(), which makes the code
simpler and more robust.

Test Plan: Ran the tests

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, stash, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4180874

Signature: t1:4180874:1479286125:bbd2a36aa86237d68036b9d7b0a580829219d869
2016-11-16 12:11:06 -08:00
Jun Wu
ca10807a38 fbamend: fix Python 2.6 syntax 2016-11-16 14:23:57 +00:00
Stanislau Hlebik
fd8f0625ef infinitepush: add hg backup command
Summary:
Backups all new non-extinct [1] commits to bundlestore.

When it is called for the first time `hg backup` will backup all draft
visible commits. Next backups will save only new commits since the last
backup (it is recorded in `.hg/store/infinitepushbackuptip`[2]).

It's an initial implementation. Later the following features will be added:
1) It will be called automatically whenever user creates or strips commits
or even creates or deletes bookmarks
2) It will also save all local bookmarks and all local heads
(probably only visible).

Note: calling `pushcmd` directly does not set default values for opts. That means that `--to` will be None and `_scratchbranchmatcher` will throw exception. Let's add a check to ensure that `--to` is never None.

[1] I also want to backup extinct commits (i.e. obsolete invisible commits).
But it will require bigger changes in discovery algorithm, so I'd leave it
for later.
[2] The name is a bit verbose. But I want to keep `infinitepush` part to make
it easier to debug problems.

Test Plan: Run `test-infinitepush-*`

Reviewers: rmcelroy, mitrandir, durham

Reviewed By: durham

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4147651

Tasks: 12479677

Signature: t1:4147651:1479229440:3eb38880c14f18e9a2fb4eaba44bedf079bca506
2016-11-16 02:13:17 -08:00
Arun Kulshreshtha
9e77a16c75 fbamend: remove old implementation of hg next --rebase
Summary: Remove the old _nextrebase() function. This function is no longer used after the updates to `hg previous` and `hg next`.

Test Plan: All unit tests pass. This code is not called anywhere so removing it should be a no-op.

Reviewers: quark, #sourcecontrol, durham

Reviewed By: durham

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4121784

Tasks: 14119420

Signature: t1:4121784:1478305767:15a737545d0e85c175018e190cb92f722ebad7d8
2016-11-15 18:46:59 -08:00
Arun Kulshreshtha
97ed6a8afa fbamend: add new implementations of hg previous and hg next
Summary:
This diff replaces the `hg previous` and `hg next` commands from the evolve extension with new implementations. The new commands have several features not found in evolve:

  - Users can now move by multiple commits at a time. Example: `hg next 3`
  - Users can move all the way to the top or bottom of the current stack with the following new flags: `hg next --top` and `hg prev --bottom`
  - Users can rebase child commits on demand with `hg next --rebase`. This flag can be combined with the others to specify exactly how many commits to rebase. Example: `hg next --rebase 4` (move 4 commits up the stack, rebasing as needed) `hg next --rebase --top` (rebase all the way to the top of the stack)
  - When a user lands on a particular commit, if there is a bookmark it will be automatically activated. The user can also move to the previous or next commit with a bookmark with the `--bookmark` flag.
  - The user can pass the `--newest` flag to resolve ambiguity in situations where history is nonlinear. With the flag the command will always choose the commit with the higher rev number at each step.
  - The commands now show commit hashes instead of rev numbers when showing the user what commit they landed on.

Apologies in advance for the large diff. I've tried splitting it up as best as I could, but the unit tests introduced here significantly increase the size. I figured the unit tests should be part of this diff instead of being added in a separate one. This diff is almost entirely new code, so I hope that helps a little bit. Let me know if there's a better way to split this up.

Test Plan: See the new unit tests for an extensive look at the new commands in action. Essentially, all of the features mentioned above should work as expected.

Reviewers: quark, #sourcecontrol, durham

Reviewed By: durham

Subscribers: stash, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4121778

Tasks: 14119420

Signature: t1:4121778:1478774243:a7131593222bc329f541e77e1d3ebd8222e47e79
2016-11-15 18:46:56 -08:00
Arun Kulshreshtha
79ca27552d fbamend: make inhibit extension's post-transaction callback fire after rebase
Summary: The new `hg next` and `hg previous` implementations rely on the inhibit extension to inhibit visible obsolete changesets, thereby preventing unstable descendants from appearing as "unstable" in smartlog for end users. The inhibit extension does this by registering a callback that runs whenever a transaction closes. However, it turns out that this callback isn't being called at the end of a rebase, so we need to add a transaction to call it ourselves.

Test Plan: All unit tests for fbamend still pass. Currently, there isn't a way to demonstrate the issue that occurs if this code isn't here; the new `hg previous` and `hg next` functions introduced in the next diff in this stack do provide a unit test to check for this behavior.

Reviewers: quark, #sourcecontrol, durham

Reviewed By: durham

Subscribers: rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4121768

Tasks: 14119420

Signature: t1:4121768:1479264251:841bd6473df810e3d663550ef48ec6948476f3a2
2016-11-15 18:46:52 -08:00
Tony Tung
525b3024a2 cdatapack: clear the counter after we madvise away
Test Plan: visual inspection.

Reviewers: durham

Reviewed By: durham

Subscribers: mitrandir, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4185510

Tasks: 14574117

Signature: t1:4185510:1479252916:6477e9c1708e6bf74fa3c2668ce222597ab4c8ae
2016-11-15 18:19:08 -06:00
Jun Wu
127bd207b5 linkrevcache: wrap _adjustlinkrev to take advantage of the linkrev database
Summary:
Wrap `_adjustlinkrev` so it first tries the possibly faster path (linkrev
database). Then fallback to the original linkrev algorithm.

Also fixes an issue that `{k : v ...}` syntax does not work in Python 2.6.

Test Plan: Added a very simple test

Reviewers: #sourcecontrol, durham

Reviewed By: durham

Subscribers: durham, stash, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4143315

Signature: t1:4143315:1479234110:07e87c7584e0cfd627b3f9e2fae1a2a34beb390f
2016-11-14 18:04:29 +00:00
Wez Furlong
9b5499d572 phabdiff: don't require a space when parsing the Tasks field
Summary: Allow for any number of spaces after the colon

Test Plan: arc unit

Reviewers: #sourcecontrol

Subscribers: markisaa

Differential Revision: https://phabricator.intern.facebook.com/D4179619

Tasks: 14532985
2016-11-15 12:28:21 -08:00
Jun Wu
6860161db4 linkrevcache: add a command to verify the linkrev database
Summary:
Adds a `debugverifylinkrevcache` to verify the linkrevs in the database,
against the vanilla adjustlinkrev algorithm.

Test Plan: A simple test is added. The code is also manually tested on my hg repo.

Reviewers: #sourcecontrol, stash

Reviewed By: stash

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4143240

Signature: t1:4143240:1479217639:bb9b5814b405b9e60a0722b01fc2c918c5a24b35
2016-11-14 18:04:29 +00:00
Jun Wu
6864d82976 linkrevcache: add a command to build the linkrev database
Summary:
This diff implements a `debugbuildlinkrevcache` command which builds the
linkrev database.

Test Plan: A simple test will be added when the cache can be verified.

Reviewers: #sourcecontrol, stash

Reviewed By: stash

Subscribers: stash, jsgf, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4143200

Signature: t1:4143200:1479131313:858042c86b5894898333900f63ea0cd50c47e934
2016-11-14 18:10:30 +00:00
Jun Wu
de07b8a110 linkrevcache: attach the linkrev database to the repo object
Summary: This diff makes the linkrev database accessible via `repo._linkrevcache`.

Test Plan: A test will be added when the feature is more complete.

Reviewers: #sourcecontrol, jeroenv

Reviewed By: jeroenv

Subscribers: jeroenv, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4143175

Signature: t1:4143175:1478558825:2c0eb0fa0f77cf004aee7c40ffc2a6ea417f6212
2016-11-14 18:08:30 +00:00
Jun Wu
dd91b969e6 linkrevcache: implement the database
Summary:
This diff implements some simple linkrev database classes which uses dbm to
store linkrev candidates.

Why not sqlite? sqlite's insertion seems to be much slower with index being
present, so we have to do manual buffering and do insertion in batch, not to
say parsing SQL itself can introduce overhead. If we really want sqlite for
some reason (i.e. its support for concurrent writer and reader seems to be
better), we can still implement it as a dbm engine.

Test Plan: A test will be added when there is something usable.

Reviewers: #sourcecontrol, stash

Reviewed By: stash

Subscribers: jsgf, stash, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4143119

Signature: t1:4143119:1479131795:5f6e368ebd1798b874cec28ac05b137eb3bea86e
2016-11-14 18:03:15 +00:00
Arun Kulshreshtha
1c6b719130 fbamend: make restack start from bottom of the stack
Summary:
Previously, `hg rebase --restack` would find the closest stable base commit with unstable descendants on its precursors, and begin the restack operation from this commit. What this meant is that it would not resolve all instability in the current stack -- only instability above the first instance of instability detected by searching downward from the current changeset.

Given that we will soon be able to perform local fixup operations with something like `hg next --rebase --top`, it no longer makes sense to keep `hg rebase --restack` as a local operation. Instead, it should resolve all instability in the current stack. This diff changes the behavior to always begin restacking from the bottom of the stack.

One thing to note about this diff is that it completely removes divergence detection from restack. This is because in the process of making this change, I discovered a bug in the original version of restack whereby divergence would only be detected if it involved a precursor of the base changeset. This is because the check was only happening in _latest(), which after several rounds of changes to this code, was only being used in _findstablebase(). In all other cases, the implicit behavior was that unstable children are rebased onto the newest visible successor of their parent, as determined by rev number.

It turns out that altering this behavior to skip rebasing unstable descendants of divergent changesets is nontrivial, and furthermore, one could argue that the current behavior is actually desirable. (Furthermore, this change removes the O(n^2) behavior from _latest().) Regardless of whether it is desirable or not, this diff makes that implicit behavior explicit, and modifies the tests to explicitly check for the new behavior. If we do decide to skip divergent changesets, this will need to happen in another diff as it will require some restructuring of the restack target finding logic.

Test Plan:
See unit tests.

1. Create a stack of commits.
2. Pick several commits in the stack, and amend them without the `--rebase` flag, thereby leaving their respective descendants unrebased.
3. Update to the top of the stack.
4. Run `hg rebase --restack`. All instability will be resolved.

Reviewers: quark, #sourcecontrol, durham

Reviewed By: durham

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4144899

Tasks: 14041544

Signature: t1:4144899:1478781779:1a94dbc1c6b69d7116f1d40593a83684ed44942a
2016-11-10 11:25:19 -08:00
Stanislau Hlebik
1b8b5987b2 infinitepush: overwrite bundle for old revs
Summary:
Previously if bundle has contained nodes that are already saved in bundlestore
then we wouldn't update bundleid value for these nodes.
That means that we can never delete a bundle because there will always be a
node that points to it.

I suggest to always update bundleid value for all nodes in the bundle.

Test Plan: Run `test-infinitepush-*`

Reviewers: durham, mitrandir, rmcelroy

Reviewed By: rmcelroy

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4130662

Tasks: 12479677

Signature: t1:4130662:1478565392:682fe39c5832d5edc84c84541f0ba6c1f2e1daf8
2016-11-08 02:27:25 -08:00
Jun Wu
77d215b481 remotefilelog: check file existence in _revertprefetch
Summary: This is to address a crash report: P56804194.

Test Plan: `arc unit`

Reviewers: durham, #sourcecontrol, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4141002

Signature: t1:4141002:1478547571:2a06a48c3687ce91f6a52b59ddc01a17466782a8
2016-11-07 18:30:10 +00:00
Jun Wu
855af710bb fastannotate: support remotefilelog
Summary:
In order to support remotefilelog, do not construct
`mercurial.context.context` object directly. This reverts a previous attempt
to use the fastest manifest look up - `ctx[path]` actually seems good
enough.

A minor change is made in `protocol.py` to ignore failed requests.

Test Plan: Run existing tests

Reviewers: #sourcecontrol, rmcelroy

Reviewed By: rmcelroy

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4127025

Signature: t1:4127025:1478541481:c15477332582e2c9b22e2732e9e76bf37b5fba74
2016-11-03 20:55:16 +00:00
Jun Wu
1e067cbd84 fastannotate: add an option to use flock
Summary:
When building the annotate cache server-side in batch, the current lock
mechanism creates and deletes small files very frequently and is not friendly
to the filesystem.

This diff introduces a `flock` approach which does not write to the disk and is
more friendly to the filesystem. The downside is, some network filesystem won't
support the lock correctly.

Since it seems to be a big improvement, especially at server-side, it's turned
on by default when it is supported by the operating system.

The documentation of config options have been updated so important options
appear first.

Test Plan:
Run the existing tests as the flock is turned on by default on my system.


Reviewers: #sourcecontrol, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4103941

Signature: t1:4103941:1478541436:b4877df397ae2f0d58a611f13917704850fd62c1
2016-10-30 02:13:07 +00:00
Stanislau Hlebik
e61c86753e infinitepush: add tests for sqlindexapi
Test Plan: Run `test-infinitepush-*`

Reviewers: durham, mitrandir, rmcelroy

Reviewed By: rmcelroy

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4124846

Tasks: 12479677

Signature: t1:4124846:1478537233:afb783ba195fd8f0d6e55d0fb8666220705575cd
2016-11-07 12:35:34 -08:00
Arun Kulshreshtha
8afd0a8a75 fbamend: add helper functions for hg previous and hg next
Summary: Add some miscellaneous new helper functions that will be used in the new implementation of `hg previous` and `hg next`. These are just utility functions for things like setting bookmarks, printing out changeset lists, and hiding command line options. Also fixed a few typos in comments.

Test Plan: All tests still pass. This change should be a no-op as these functions are not called anywhere.

Reviewers: durham, quark, #sourcecontrol, stash

Reviewed By: stash

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4121764

Tasks: 14119420

Signature: t1:4121764:1478162841:32362fef3727e67e46b5700d9242be5c0cd62a52
2016-11-03 11:11:41 -07:00
Arun Kulshreshtha
2e9aba5940 fbamend: rename/refactor wrapper functions
Summary: Do a minor refactor and rename of some of the wrapper functions in this file. There are no functional changes. These names were chosen to follow a consistent convention and be less confusing when taken together with the new `hg previous` and `hg next` functions that will be added in a later diff.

Test Plan: All tests still pass.

Reviewers: durham, quark, stash

Reviewed By: stash

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4121755

Tasks: 14119420

Signature: t1:4121755:1478164041:d8c603c32554d1d7c1df9706386c015f90ad62c6
2016-11-03 11:11:39 -07:00
Arun Kulshreshtha
cdfe81c7d3 fbamend: change test for restack to use globs for rev numbers
Summary: One of the new features I'm adding to `hg previous` and `hg next` is replacing the rev numbers with commit hashes, since these are more familiar and intuitive to users. As a first step, update this test to glob out the rev numbers.

Test Plan: Test still passes.

Reviewers: durham, quark, #sourcecontrol, stash

Reviewed By: stash

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4121748

Tasks: 14119420

Signature: t1:4121748:1478162085:c7bc64f133dcfd7af67f349a10c0e0a587a8aa19
2016-11-03 11:11:37 -07:00
Jun Wu
5f83e80668 fastannotate: add a debug command to update cache
Summary:
The `debugbuildannotatecache` is to pre-populate the annotate cache
server-side, or pre-fetch the annotate cache client-side.

To avoid import cycle, `cmdtable` is moved from `__init__.py` to
`commands.py`.

Test Plan: Modified existing tests.

Reviewers: #sourcecontrol, stash

Reviewed By: stash

Subscribers: stash, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4103932

Signature: t1:4103932:1478094621:e82192a12fe4b0b3ac732dbe4c127a2e93fc6b4c
2016-11-01 15:10:21 +00:00
Jun Wu
93c918c92e absorb: fix an issue where changesets do not get deleted correctly
Summary:
The old code didn't test if the new commit will become empty correctly, if its
"memworkingcopy" contains changes inherited from its parent. This diff fixes
the problem.

Thanks @stash for discovering the issue!

Test Plan: Added a test case for this issue. Make sure the old code cannot pass.

Reviewers: #mercurial, stash

Reviewed By: stash

Subscribers: mjpieters, stash

Differential Revision: https://phabricator.intern.facebook.com/D4113438

Signature: t1:4113438:1478077373:194104b902bdfe6827858c2916a7f3f0cd83ee6c
2016-11-01 21:16:13 +00:00
Jun Wu
ecbfc04760 fastannotate: improve documentation of the command
Summary:
The command docstring was written with a full repo in mind. With a shallow
repo, things become different. This diff updates the documentation so it
also makes sense for a shallow repo.

Test Plan: Vim spellchecker.

Reviewers: #sourcecontrol, stash

Reviewed By: stash

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4098257

Signature: t1:4098257:1477920765:8a48b332e786c4c538e939536564262214b3cc04
2016-10-28 19:49:59 +01:00
Jun Wu
21bbbebffe fastannotate: implement downloading logic for clients
Summary:
This diff makes the client `fastannotate.client=1`  download
linelogs and revmaps from the server on demand.

Test Plan: Added a new test

Reviewers: #sourcecontrol, stash

Reviewed By: stash

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4103929

Signature: t1:4103929:1477997846:608cf58c090806dafa21ae01feab90a6552100fc
2016-10-29 23:44:34 +01:00
Jun Wu
ca37364c8d fastannotate: implement a simple wireprotocol transferring the annotate cache
Summary:
This diff adds a simple wireprotocol that the client can download fastannotate
cache files from the server.

Design decisions:
- Transfer full files. The linelog is not append-only and rsync-like
  incremental updates is not trivial. Since the files look small (usually <
  1MB), send full copies for now. If file size becomes an issue, we can
  improve it later.
- The server can send arbitrary number of files with paths defined. This
  provides the flexibility to send extra files without changing the protocol.
  For example, if we want to speed up `--deleted`, we may want to download
  `linemap` files.
- No compression. The revmap is basically not compress-able because it
  contains the hashes. The linelog could be compressed though, but they are
  small enough, and we may rely on compression on other layers, like SSH.
- `lastnode` check - if the last node of revmap the client reports matches
  what the server has, the server will assume it's up-to-date and do nothing.

Note that this diff is only about the protocol and methods on peer, localrepo
objects. The client-side command is not aware of the feature yet.


Test Plan:
- Clone from a local repo A to B, via the ssh protocol.
- Config both repos so B is the client, A is the server.
- Run `hg debugshell --debug` from B, make sure the client can get contents
  correctly (check the file contents from both the client and the server),
  also make sure the non-existed files will be ignored:

  In [1]: repo.prefetchfastannotate(['README', 'Makefile', 'NOT-EXIStED'])
  running ssh localhost ...
  sending hello command
  sending between command
  remote: 383
  remote: capabilities: .... getannotate
  remote: 1
  fastannotate: requesting 2 files
  sending batch command
  fastannotate: server returned
  fastannotate: writing 4976 bytes to fastannotate/default/README.l
  fastannotate: writing 813 bytes to fastannotate/default/README.m
  fastannotate: writing 9024 bytes to fastannotate/default/Makefile.l
  fastannotate: writing 2453 bytes to fastannotate/default/Makefile.m

- Make sure the second time fetching these files, they will be skipped because
  of the uptodate check:

  In [2]: repo.prefetchfastannotate(['README', 'Makefile', 'NOT-EXIStED'])

- Manually create a diverge case where the client and the server disagrees
  about what "master" (or "mainbranch") is. Make sure that although the
  request will be send by the client, the server can return nothing because of
  the lastnode check.


Reviewers: #mercurial, durham, stash

Reviewed By: stash

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4063249

Signature: t1:4063249:1477920081:9bfab3031aa3118060c433f6e27c5b32aa25040c
2016-10-29 22:45:20 +01:00
Jun Wu
d8dd5ef6e8 fastannotate: add some helper methods to annotatecontext
Summary:
As we are going to add a protocol to transfer fastannotate cache over the
network, we need some basic methods to test if the cache is up-to-date.
Namely, this diff adds two methods:

- lastnode: used by isuptodate
- isuptodate: useful for both the server and the client to test if the cache
  needs update or re-download

Test Plan:
These methods will be used later and a `.t` will be added when a fastannotate
client can download annotate cache from the server.

Reviewers: #sourcecontrol, stash

Reviewed By: stash

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4103906

Signature: t1:4103906:1477997881:ad6843fdb683a550561758cd5babc56622f2ead3
2016-10-29 20:24:48 +01:00
Jun Wu
ba972813b3 fastannotate: load linelog and revmap lazily
Summary:
Previously, when using `with annotatecontext(...) as actx:`, the code will
construct the revmap and linelog objects, unconditionally.

Sometimes the caller just want to know some cheaper information, like the
revmap size, or the last hash of the revmap. It's unnecessary to construct the
revmap or linelog objects, while annotatecontext's locking could be useful.

This patch makes revmap and linelog reading lazily loaded so callers can use
`with annotatecontext(...) as actx` in more cases without worrying about
performance.

Test Plan: `arc unit`

Reviewers: #sourcecontrol, stash

Reviewed By: stash

Subscribers: stash, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4103902

Signature: t1:4103902:1477997160:8638d8c772d326a5ad2a61124c96727ed6c3a87b
2016-10-29 19:45:16 +01:00
Jun Wu
57890ff99f fastannotate: add a method to quickly get the last node of the revmap
Summary:
Previously, to get changeset hashes in the revmap, we have to construct
the `revmap` object, which reads all content of a revmap.

In the following patches, we want to quickly get the last hash of the
revmap to test if the linelog / revmap is up-to-date or not. This patch
adds a method to do so.

Test Plan: Modified existing tests.

Reviewers: #sourcecontrol, stash

Reviewed By: stash

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4103898

Signature: t1:4103898:1477988583:15869dfe9028068563b03d1f06d2337562049751
2016-10-29 19:19:18 +01:00
Stanislau Hlebik
f4f6b2411a infinitepush: inhibit pulled revs
Summary:
Pulling scratch commits that were stripped before results in
abort: 00changelog.i@c153241c1ea22cbce8a229dccdf7f46de827d962: filtered node!

The reason is that upstream pull does
checkout = str(repo.changelog.rev(checkout))

and it fails if revision is filtered.
I suggest to inhibit filtered revisions that are present in the local repo.
It makes sense because if user tries to pull scratch branch he or she
probably expects to see them in the repo even though these revisions were
stripped before.

Also we can avoid doing any pull if all revisions are present in the repo.
I decided not to do this because it complicates code (for example,
we need to ensure that bookmarks are updated correctly) and it doesn't give
any significant benefits.

Test Plan: arc unit

Reviewers: durham, rmcelroy, mitrandir

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4103869

Tasks: 12479677
2016-11-01 03:42:26 -07:00
Stanislau Hlebik
acdd58a41e infinitepush: possible fix to prevent scratch bookmarks disappearing
Summary:
Sometimes remote scratch bookmarks disappear during `hg pull`.
I can't reliably reproduce it, but the best guess I had so far is
that `repo.names['remotebookmarks']` is not up-to-date.
Let's add `clearnames()` before.

Test Plan: Run `test-infinitepush-*`

Reviewers: rmcelroy, mitrandir, durham

Reviewed By: durham

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4104227

Tasks: 12479677

Signature: t1:4104227:1477954723:70e1ab2b4a90f26046a1dd05e5bb4aab68777e65
2016-11-01 01:53:41 -07:00
Stanislau Hlebik
cf1214923e infinitepush: disable pushes to svn repo
Summary:
During infinitepush test we noticed that `hg push -r . --to SCRATCHBRANCH`
fails with stack trace in www. This is expected because infinitepush doesn't work
with svn servers. But it's better to print a nice error message instead of stack trace.

Test Plan:
Try to push to scratch bookmark from www without specifying path:
stash@dev1918 ~/www (344c6bf) $ hg push -r . --to hack/durhamstash
abort: infinite push does not work with svn repo
(Did you forget to `hg push default`?)
[Exit: 255]

Got nice error instead of stack trace.

Push to scratch bookmark from www with path specified works fine

Reviewers: rmcelroy, mitrandir, durham

Reviewed By: durham

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4103745

Tasks: 12479677

Signature: t1:4103745:1477950448:14b5ae8484f3e65cb88c83b2628f73f3a670b1a0
2016-11-01 01:50:34 -07:00
Stanislau Hlebik
563a57128e infinitepush: add mysql waittimeout server-side
Summary:
Saw a few timeout errors during infinitepush test.
Previous timeout was 10 secs, it makes sense to increase it.

There was a unit test failures because of the config options.
So this diff also adds missing config docs and copyrights headers.

Test Plan: Build the package, install it locally and run integrationtests

Reviewers: durham, rmcelroy, mitrandir, andrasbelo

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4103689

Tasks: 12479677
2016-11-01 01:44:38 -07:00
Durham Goode
201d5f2149 remotefilelog: fix getchangegroup signature
Summary:
The upstream getchangegroup function changed back in August and our attempt at
fixing it here did not correclty handle the case where remotefilelog is loaded
but not enabled for this repository.

This patches fixes the signature, and makes our wrapping more resitant to future
signature changes.

Test Plan: Added a test.

Reviewers: #mercurial, jsgf

Reviewed By: jsgf

Subscribers: jsgf, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4099519

Signature: t1:4099519:1477698801:adcb406f1908c4f7e508e5f99e126f383c6f2574
2016-10-31 13:40:33 -07:00
Durham Goode
922c03630a remotefilelog: make test less flaky
Summary:
The stderr and stdout in this test has a flaky ordering issue when run in
parallel with other tests. Let's just eat the stderr since all we care about is
the stdout message from the remote and the error code.

Test Plan: Ran it

Reviewers: #mercurial, simonfar

Reviewed By: simonfar

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4106366

Signature: t1:4106366:1477946237:2e188a1abde93a06ec97d99c237851955a23f20a
2016-10-31 13:39:17 -07:00
Durham Goode
8aad6cabdd tests: glob away system specific output
ls -l prints a 'total #' line which apparently is system specific. So let's make
it a glob.
2016-10-28 16:50:27 -07:00
Jun Wu
f183d30cdf fastannotate: move path-related methods to a class
Summary:
Previously, the linelog and revmap paths are private information inside
`annotatecontext`. But we still want to get those paths in other places,
like the perfhack path check, or the quick up-to-date check (linelog vs
filelog) server-side. Let's expose the information by adding the helper
class.

Test Plan: Run existing tests

Reviewers: #mercurial, stash

Reviewed By: stash

Subscribers: stash, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4062835

Signature: t1:4062835:1477552929:7ece4f492e6b1e6f22cc626044819d5e3e2a66aa
2016-10-22 01:37:08 +01:00
Jun Wu
2698cf1d22 scripts: unify spwaning run-test logic
Summary:
Previously, lint.py and unit.py have different logic spawning
run-tests.py.

The logic in `unit.py` is more robust: it sets `PYTHONPATH` and
`cwd`. It sets `-j` according to CPU cores. It can find `run-tests.py`
even if `MERCURIALRUNTESTS` is not set. You can run the script
from any directory (not only reporoot).

Test Plan: Run `lint.py` and `unit.py`, from the `scripts` directory and reporoot.

Reviewers: #sourcecontrol, stash

Reviewed By: stash

Subscribers: stash, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4095437

Signature: t1:4095437:1477663516:13a7ac4270435c272077132915f9d02cc98a5afb
2016-10-28 13:58:48 +01:00
Durham Goode
df79de6e69 treemanifest: add test for commit producing trees
Summary: Adds a test for verifying that hg commit adds a tree pack to local storage.

Test Plan: Ran it

Reviewers: #mercurial, zamsden

Reviewed By: zamsden

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4082915

Signature: t1:4082915:1477590191:6637b9ab0fdad27a7d4933934168b57620dee235
2016-10-27 15:25:34 -07:00
Durham Goode
7ae5ac490d treemanifest: add test for autocreatetrees
Summary:
Adds a simple test for checking that trees are created during pull when
autocreatetrees is enabled.

Test Plan: Ran it

Reviewers: #mercurial, zamsden

Reviewed By: zamsden

Subscribers: zamsden, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4082871

Signature: t1:4082871:1477590276:09e594f4b87628bae8654ffdfc0bba18beb7ccad
2016-10-27 15:25:32 -07:00
Durham Goode
7f711f8f84 test: rename ctreemanifest test
Summary:
Since ctreemanifest is no longer in remotefilelog, let's rename the test to drop
the remotefilelog reference.

Test Plan: Ran it and check code

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4082288

Signature: t1:4082288:1477506197:a1f591b98a087ad838fef5cc0a17e8352b6bc6e3
2016-10-27 15:25:29 -07:00
Stanislau Hlebik
8b420cd314 fastmanifest: fallback to old manifest in readshallowfast
Summary:
fastmanifest makes bundling slower because fastmanifest `readshallowfast` always returns full manifest.
That's a big problem for infinitepush.
Let's copy-paste readshallowfast implementation from upstream. It uses readshallowdelta() if possible.

Test Plan:
1) Run all the tests for fb-hgext
2) Run infinitepush with this extension enabled. Make it is fast

Reviewers: durham, simonfar, rmcelroy, quark

Reviewed By: quark

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4088360

Tasks: 13907166

Signature: t1:4088360:1477580931:746e4054380403abbc52d1922583021b81f31bb6
2016-10-27 09:23:05 -07:00
Jun Wu
79b9423cee fastannotate: improve compatibility with older hg
Summary:
Before ba91edc29ce5, `diffopts` does not have `__dict__`. Let's fix that by
querying keys from `diffopts.default`, instead of using `__dict__`.

Test Plan: `arc unit`

Reviewers: #sourcecontrol, stash

Reviewed By: stash

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4086921

Signature: t1:4086921:1477554146:e48647c59769c2951518b091350289aa966aab7b
2016-10-27 00:19:05 +01:00