Summary:
Previously treemanifest and fastmanifest were largely unaware of each other. If
the fastmanifest was available, we'd use that. If not, we'd try tree. Then we'd
fall back to flat. When comparing two manifests, this could cause problems since
if one manifest was fast and one was tree, we'd have to fall all the way back to
flat manifests to compare them.
This patch adds the ability to build a treemanifest from a fastmanifest for
inmemory manifests that were copied from a commit that has both a fastmanifest
and a treemanifest. We do it by diff'ing the inmemory fastmanifest with the
commit fastmanifest, then applying that diff to the commit's treemanifest. Then
we can compare that tree with the other tree.
This is particularly useful when doing things like 'hg up master' after a pull.
The new master commit probably has a treemanifest, but doesn't yet have a
fastmanifest, while the commit the user is on probably has both. Now we can do
that hg up without having to parse a full manifest.
Once treemanifest has been enabled, every commit (both pulled and user created)
should have treemanifests. Therefore this case should never happen when
comparing two normal commits. So in theory, the only way to hit this case is
when Mercurial does a manifest.copy() on a manifest that has a fastmanifest
(like when reading a workingctx which must copy the commit manifest and apply
working copy changes), since the copy will choose to copy the fastmanifest and
leave the treemanifest behind. This patch addresses that case, so I think there
shouldn't be other cases.
Test Plan: Adds a test that covers this case.
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4402604
Signature: t1:4402604:1484216866:1c075f314026aaf608095b4aca37c9a4277ababa
Summary:
We have run into some cases where users ended up with empty pack file in their
packs directory. Just log a warning in this case and skip this pack file,
rather than letting the exception propagate up and crashing the command.
Test Plan:
Created empty 0000000000000000000000000000000000000000.histpack and
0000000000000000000000000000000000000000.histidx files in my repository's
hgcache directory, and confirmed that "hg log" now simply warns about them
instead of crashing.
I didn't really test the perftest.py or treemanifest_correctness.py extensions
much. They seem to throw exceptions, and look like they have maybe gotten a
bit stale. I fixed one minor typo, but didn't dig into the other exceptions
too much.
Reviewers: durham
Reviewed By: durham
Subscribers: net-systems-diffs@, yogeshwer, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4402516
Tasks: 15428659
Signature: t1:4402516:1484155254:96d2980efcec2d735257b08910e1ca437ef1dad6
Summary: I somehow missed this test when updating all of them with fbamend's new error message.
Test Plan: Test now passes.
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4399110
Signature: t1:4399110:1484094504:2c7a8124849c9fd32fb57162dfb1ba7a90b1efbc
Previously we attached the original node to the copy of the manifest we made.
This is almost always incorrect. In the main case, we make a copy of the p1
manifest during commit and then edit that manifest to form the new manifest. By
storing the node (and never invalidating it), there's the opportunity for the
code to accidentally load the old p1 manifest instead of the in memory one.
The fix is to not pass node to the copy. This affects the tests because some
tests relied on the copied manifest being able to access the original tree/cache
later. It just happened to work because we never modified the manifest before
performing those loads.
This is similar to the patch to matches() that made the match result not have
node either.
This patch also cleans up the fastdelta logic that attempted to read the
_treemanifest directly, instead of going through the normal read path. This is
what actually caught the bug, since it read the treemanifest from disk, while
the in memory cachedmanifest had changes.
Previously, when fastmanifest.usecache was set to false, we just had the cache
return False every time we checked if something was in the cache. Unfortunately,
this didn't prevent us from constructing new inmemory fastmanifestdicts, which
didn't play nicely with treemanifest.
Let's make usecache do the exact same thing as usetree, which means storing
'False' in the __cachedmanifest field, and using that to indicate that field is
completely disabled.
There were two openers in use in this code: opener and origself.opener. 'opener'
was only being created when it was the first manifest in the transaction, so the
line at the bottom that depended on 'opener' existing would fail if a second
manifest was created during the transaction (like during an amend). We just need
to fix the last line to use the correct origself.opener, and fix the 'opener'
name to be less accidentaly reusable.
The _treemanifest function was hard coded to return None if self.node was None,
even if self.__treemanifest was present. A future patch removes self.node in
many cases, so this ended up breaking since it could no longer return the
treemanifest.
Summary:
The remotenames extension automatically adds a --dest option to `hg rebase`, which causes
restack to abort. Instead, we should just drop the 'dest' key and proceed.
Test Plan: Attempt to use `hg rebase --restack` in fbsource when the current bookmark is tracking a remote bookmark.
Reviewers: #mercurial, durham, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4399846
Tasks: 15422257
Signature: t1:4399846:1484084585:4a6ec907032e7d8b1388980870b6d72cb560829f
Summary: Adding progress bar to slow prefetch process.
Test Plan: Perform a prefetch with :-10 or :-100 and see a progress bar displayed.
Reviewers: hanw, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters, hanw
Differential Revision: https://phabricator.intern.facebook.com/D4398078
Tasks: 9993344
Signature: t1:4398078:1484071212:f906493464340bea103be66133aa7026284143c7
Summary: Upstream Mercurial now has native support for statprof, rendering this file useless.
Test Plan: Remove statprofext.py* from my installed hg, confirm it still works. Build fb-hgext and run tests, confirm no new failures
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4397926
Signature: t1:4397926:1484066120:0c1469d838a91a65fb57ecc858675256087e2c48
Summary: Added some tests for this also
Test Plan: arc unit in fb-hgext
Reviewers: #sourcecontrol, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4397881
Tasks: 15360841
Signature: t1:4397881:1484066348:1b9fb24a30a9962de0dad858f4d57956a2c91dc4
Summary: To encourage people to learn about and use `hg restack` (which is a more general fixup tool), tell users about it instead of `hg amend --fixup`. I wasn't sure whether to advertise it as `hg rebase --restack` or just `hg restack`, so I've included both.
Test Plan: Do an amend in the middle of a stack and observe the new wording. Unit tests reflect changed wording.
Reviewers: durham, #sourcecontrol, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4390111
Signature: t1:4390111:1483962208:46951c8f5e4c499840b0b4b0ee8ac1e1b6c14ca9
Summary:
On a big repo with many old heads bundle that is sent to the server can become
big (in some cases even 500 Mb). That looks like a waste of bundlestore space
and will probably make backup and restore slower. Most of the space is taken
by manifest deltas because it prefers to diff manifest against previous commit
in the bundle. There are two possible approaches to reduce the size:
1) Send many small bundles (for example, one bundle per head)
2) Wrap deltaparent function and diff against actual parent previous commit in the bundle.
I chose the second approach for the following reasons:
1) It's easier to implement (main reason)
2) Many bundles probably means slower restore because there will be many requests to the bundlestore instead of just one
With this diff bundle size was reduced from 500 Mb to 8 Mb.
It can potentially increase CPU usage. I'm not sure how bad is it and will investigate it more.
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: simpkins, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4384053
Tasks: 12479677
Signature: t1:4384053:1483664446:62ec30fad433e8d279758926199a8330cb73ed2b
Summary:
If client pulled a commit without filelogs (because of remotefilelog) and
this commit was later stripped on the server, then attempt to backup this
commit fails because of missing filelogs. It's a rare issue but unfortunately
it happens sometimes. To fix it let's exclude from the backup all the
commits and their descendants that doesn't have all necessary filelogs.
Test Plan: Run infinitepush tests
Reviewers: mitrandir, rmcelroy, durham
Reviewed By: rmcelroy
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4365747
Signature: t1:4365747:1483466164:560cdf5cd2369cd2603dfd0fe6b30d7a70951f00
Summary:
Test was flakey because wait_for_background_backup.py waited for change in
backup state file but then logfile was checked. In this case it was possible
that backup state file was already synced while logfile was not.
Let's not use logfile at all and just check backup file.
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4383660
Tasks: 12479677
Signature: t1:4383660:1483727504:2d738ef5ea8d016a8df71cbc6a99d44f24a77f3e
Summary:
I got the following profile results when profiling smartlog:
{P56936293}
It suggests that regex compiling takes too much time. I tend to think that
profiling is inaccurate in this case because with this diff I didn't get any
noticable speed up. But this diff won't do any harm.
Test Plan: arc unit
Reviewers: #sourcecontrol, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4387265
Signature: t1:4387265:1483724595:43de3c694109be2d4343d8ebdbc7ab79aa9edb04
Summary:
`pushbackup` command became huge. This diff splits it into a few
smaller functions
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4384008
Tasks: 12479677
Signature: t1:4384008:1483659549:a97b63a38b702a55d19cf6b47a6fd6b2547b9168
Summary: __init__.py became huge and I suggest to split it
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4383661
Tasks: 12479677
Signature: t1:4383661:1483659134:eaf3d7e217633895a65ad568831d9c21ea0f18d8
The sshaskpass is not only a mercurial extension, but also a script that
runs separately, in which case it cannot import mercurial modules. So let's
fallback to os.environ.
Summary:
wait_for_background_backup.py does sleep internally and stops as soon as file
is present (with 0.1 sec precision). Additional sleeps shouldn't be necessary
Test Plan:
Run `test-infinitepush-*` on devserver
PYTHONPATH=/opt/facebook/hg/lib/python2.7/site-packages/:/opt/homebrew/lib/python2.7/site-packages/ /opt/homebrew/opt/python27/bin/python2.7 ../../hg-crew/tests/run-tests.py
on mac
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4381211
Signature: t1:4381211:1483551806:bbefc6632d7aed83f4a1caf8ad2ec5ba2b8f12d7
Summary:
`pushbackup` failed when there are hidden heads in the repo but no visible
heads (see test for example). This diff fixes it.
Test Plan: Run `test-infinitepush-*`
Reviewers: durham, mitrandir, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4365677
Tasks: 12479677
Signature: t1:4365677:1483465909:fd1d96211909d3eeccab4fd17a364af4a1997f85
Summary:
Recently I had this scm prompt:
{F64910275}
This is not user friendly. I suggest to output at most one remote bookmark
Also this diff copies `seq.py` from upstream hg repo
Test Plan: Run scm prompt tests on devserver and mac and with '--shell=zsh'
Reviewers: rmcelroy
Reviewed By: rmcelroy
Subscribers: simonfar, wez, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4197242
Signature: t1:4197242:1483466446:cd1fe943dc99fde0600a4e20994fa4b0f971f72a
Summary:
The '&>>' operator only works in linux, not OSX. So let's replace it with a more
verbose equivalent. 'man bash' says these two are equivalent.
Test Plan:
Running the tests normally on OSX don't actually catch this, because
this particular test is skipped because it doesn't see that evolve is installed.
We need to run the tests with:
PYTHONPATH=/opt/facebook/hg/lib/python2.7/site-packages/:/opt/homebrew/lib/python2.7/site-packages/ /opt/homebrew/opt/python27/bin/python2.7 ../../facebook-hg-rpms/hg-crew/tests/run-tests.py test-infinitepush-backup.t
To get coverage. And the test is now passing on OSX and linux.
Reviewers: stash, #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4379593
Signature: t1:4379593:1483488383:ce8a045bbc29d719eafa372db56453b83f1d8df8
Summary:
Some recent commits introduced non-OSX compatible test output. This patch fixes
that.
- 'ls' has different output text and error codes for missing files on osx
- 'sync' just waits for an in memory write to be flushed to disk, which
shouldn't actually be visible to any user land process, so it seems there's a
deeper race condition here, like wait_for_background_backup.py not working
correctly. Adding a sleep here to work around it.
Test Plan: Ran the tests on linux and osx
Reviewers: stash, #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4378618
Signature: t1:4378618:1483477739:5cd8bf82368cc417fbc7ee22410e6d962ba10342
Summary:
Without `--clean`, the user may not be able to abort cleanly.
Also, change "erase" to "discard" to be more consistent with the official
language. "changed" is replaced by "changes" as it looks like a grammar
mistake.
Test Plan: Updated existing tests
Reviewers: #sourcecontrol, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4377376
Signature: t1:4377376:1483464655:f695fd643509d7fabb88298bd1004727ee64ef76
Summary: bundle file should be closed to ensure that all the temporary files were deleted
Test Plan: arc unit
Reviewers: #sourcecontrol, ikostia
Reviewed By: ikostia
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4377148
Signature: t1:4377148:1483451990:2a1362d9e524a54ffedad8f2bc09fc622efe8ad4
Summary:
test-check-code.t was updated in core mercurial so we have to fix it in
fb-hgext
Test Plan: arc unit
Reviewers: #sourcecontrol, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4377208
Signature: t1:4377208:1483451688:bc382cdab07c6d6127d67c902458a2ccfa1373d1
Summary:
In 676596f945ea2166820ef92e692ef7fe6a6247f0 were added comments with
lines > 80. In aec81a9a80d22989bbdc8c74c1dfec9dcbbe6866 default config value
was changed.
Test Plan: arc unit
Reviewers: #sourcecontrol
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4377188
We need to keep the path in sync with the stack, and since we've already popped
the stack here, we also need to pop the path in this short-circuit case.
Previously we were relying on mfrevlog.revdiff() to produce the delta for us.
This only showed us what files were added/modified, as compared to p1, and we
had to use a heuristic to know what files were deleted (by looking at the list
of files in the commit metadata). Merge commits have a different criteria for
what is in the commit metadata (it only contains entries where the file is
different from both parents), so we can't use it for the same heuristic. So
let's fall back to a normal manifest diff for merge commits, since they are
rare.
Adds a test for verifying that conversion of merge commits into a tree works.
Summary: The --limit option was carried over from the option list for `hg log`. However, it doesn't do anything in smartlog, so this diff removes it from the option list.
Test Plan: `hg smartlog --help` no longer shows `--limit`.
Reviewers: #mercurial
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4363187
Tasks: 15186221
Summary:
Remove hard-coded devserver name and use `sync` to make sure data is written
by the background process
Test Plan: Run `test-infinitepush-*`
Reviewers: #sourcecontrol, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4361991
Tasks: 12479677
Signature: t1:4361991:1482423111:81e69044caaa608736e03cc3746cf1ff1f6625a5
Summary:
Make adding a set of rules efficient.
Currently buck autosparse writes directly to .hg/sparse, clobbering any
existing manual rules or profile settings. This is not optimal.
The command-line of hg sparse is too limited at the moment to add rules in any
other way (only one new rule per call!). The --import-rules option lets us
bulk-import and efficiently decide wether or not a refresh is needed.
Test Plan: arc unit
Reviewers: #sourcecontrol, andrasbelo
Reviewed By: andrasbelo
Subscribers: andrasbelo, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4358087
Signature: t1:4358087:1482409901:42b2d32dc624579a3d35875ba1bb126d9539cc69
Summary:
Default path was used during background backup even if non-default path was
passed on the command line. So `hg push somepath --background` is
equivalent to `hg push --background`. This is not correct and this diff fixes it
Test Plan: Run `test-infinitepush-*`
Reviewers: durham, rmcelroy, mitrandir
Reviewed By: mitrandir
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4357295
Tasks: 12479677
Signature: t1:4357295:1482330125:d2b043f8035046ab43667b4387bf642e53261681
Summary: Log active profiles for any command that uses the repository.
Test Plan: arc unit
Reviewers: #sourcecontrol, mitrandir
Reviewed By: mitrandir
Subscribers: andrasbelo, mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4346795
Signature: t1:4346795:1482251224:a9b3568eeb7e57c61273ce0dc9f4d41abcd2888e
When installed into a path like /usr/lib/python2.7/dist-packages, this file
conflicts with the hgext3rd/__init__.py from mercurial itself. Let's keep the
mercurial version the canonical one, and just skip packaging this one.
I added a warning to this repo's __init__.py about behavior divergences between
this __init__.py and mercurial's version.
Summary:
fastannotate will tell remotefilelog what nodes of a file is already known in
linelog + revmap, so remotefilelog will not prefetch those files. Previously,
fastannotate either prevents all prefetching or allows all prefetching, which
is sub-optimal.
fastannotate could now donate its sshpeer to remotefilelog, so remotefilelog
won't need to start another one (assuming they can share a same sshpeer,
could be turned off via config options). This should reduce run time if SSH
handshake is expensive.
fastannotate could now also steal sshpeer from remotefilelog, so fastannotate
won't need to start another one. Combined with the above change, there would
always be only one sshpeer shared by fastannotate and remotefilelog.
Test Plan: Modified existing tests
Reviewers: #sourcecontrol, durham
Reviewed By: durham
Subscribers: ikostia, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4325382
Signature: t1:4325382:1481933531:39d6344b2c076fbbff1f07997cd268e7cee25e80
Summary:
SCM_SAMPLING_FILEPATH is set by arcanist, and thus the tests fail when you run
arc unit.
Test Plan: Run arc unit
Reviewers: #sourcecontrol, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4342459
Signature: t1:4342459:1481929811:1ac8db74cfd86be7ad35466e6eaf2f7e35ad9140
Summary:
Logging dirstate sizes gives us a way to track sparse profile use; how big a
working copy do people actually have?
The logged info can be directed to Scuba via the sampling extension.
Test Plan: Run on a local test repo with the debugger attached. Goal
Reviewers: #sourcecontrol, quark
Reviewed By: quark
Subscribers: mitrandir, quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4340367
Signature: t1:4340367:1481920819:d7358e0bdc2cc1422b36266374344e7d2b644ae5
Summary:
Previously, if during a pushrebaes a merge commit was rebased and the
parent commit that contained the original rebase destinations content (i.e.
master's content) was on the p2 side, it would all be thrown away.
This is because the pushrebase commit works by copying the destination-repo-p1, then
applying changes from the diff of the original-repo-p1 vs original-repo-commit.
Since all the rebase destination contents was in destination-repo-p2, it is all
lost here.
The solution is to track which commit is the latest one to contain the
destination bookmark contents, and make sure it is always p1 when rebasing merge
commits. This will ensure that the destination bookmark contents are preserved
all the way up to the new commit for that bookmark.
Test Plan:
Added a test. It fails before the change and passes after. Another
test was also affected, since the p1/p2 ordering is different.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4336427
Tasks: 15092414
Signature: t1:4336427:1481881555:a22b8c363f2f7a197cef192abf34bfca926ecdc2
