Summary: Upstream Mercurial now has native support for statprof, rendering this file useless.
Test Plan: Remove statprofext.py* from my installed hg, confirm it still works. Build fb-hgext and run tests, confirm no new failures
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4397926
Signature: t1:4397926:1484066120:0c1469d838a91a65fb57ecc858675256087e2c48
Summary: Added some tests for this also
Test Plan: arc unit in fb-hgext
Reviewers: #sourcecontrol, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4397881
Tasks: 15360841
Signature: t1:4397881:1484066348:1b9fb24a30a9962de0dad858f4d57956a2c91dc4
Summary: To encourage people to learn about and use `hg restack` (which is a more general fixup tool), tell users about it instead of `hg amend --fixup`. I wasn't sure whether to advertise it as `hg rebase --restack` or just `hg restack`, so I've included both.
Test Plan: Do an amend in the middle of a stack and observe the new wording. Unit tests reflect changed wording.
Reviewers: durham, #sourcecontrol, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4390111
Signature: t1:4390111:1483962208:46951c8f5e4c499840b0b4b0ee8ac1e1b6c14ca9
Summary:
On a big repo with many old heads bundle that is sent to the server can become
big (in some cases even 500 Mb). That looks like a waste of bundlestore space
and will probably make backup and restore slower. Most of the space is taken
by manifest deltas because it prefers to diff manifest against previous commit
in the bundle. There are two possible approaches to reduce the size:
1) Send many small bundles (for example, one bundle per head)
2) Wrap deltaparent function and diff against actual parent previous commit in the bundle.
I chose the second approach for the following reasons:
1) It's easier to implement (main reason)
2) Many bundles probably means slower restore because there will be many requests to the bundlestore instead of just one
With this diff bundle size was reduced from 500 Mb to 8 Mb.
It can potentially increase CPU usage. I'm not sure how bad is it and will investigate it more.
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: simpkins, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4384053
Tasks: 12479677
Signature: t1:4384053:1483664446:62ec30fad433e8d279758926199a8330cb73ed2b
Summary:
If client pulled a commit without filelogs (because of remotefilelog) and
this commit was later stripped on the server, then attempt to backup this
commit fails because of missing filelogs. It's a rare issue but unfortunately
it happens sometimes. To fix it let's exclude from the backup all the
commits and their descendants that doesn't have all necessary filelogs.
Test Plan: Run infinitepush tests
Reviewers: mitrandir, rmcelroy, durham
Reviewed By: rmcelroy
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4365747
Signature: t1:4365747:1483466164:560cdf5cd2369cd2603dfd0fe6b30d7a70951f00
Summary:
Test was flakey because wait_for_background_backup.py waited for change in
backup state file but then logfile was checked. In this case it was possible
that backup state file was already synced while logfile was not.
Let's not use logfile at all and just check backup file.
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4383660
Tasks: 12479677
Signature: t1:4383660:1483727504:2d738ef5ea8d016a8df71cbc6a99d44f24a77f3e
Summary:
I got the following profile results when profiling smartlog:
{P56936293}
It suggests that regex compiling takes too much time. I tend to think that
profiling is inaccurate in this case because with this diff I didn't get any
noticable speed up. But this diff won't do any harm.
Test Plan: arc unit
Reviewers: #sourcecontrol, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4387265
Signature: t1:4387265:1483724595:43de3c694109be2d4343d8ebdbc7ab79aa9edb04
Summary:
`pushbackup` command became huge. This diff splits it into a few
smaller functions
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4384008
Tasks: 12479677
Signature: t1:4384008:1483659549:a97b63a38b702a55d19cf6b47a6fd6b2547b9168
Summary: __init__.py became huge and I suggest to split it
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4383661
Tasks: 12479677
Signature: t1:4383661:1483659134:eaf3d7e217633895a65ad568831d9c21ea0f18d8
The sshaskpass is not only a mercurial extension, but also a script that
runs separately, in which case it cannot import mercurial modules. So let's
fallback to os.environ.
Summary:
wait_for_background_backup.py does sleep internally and stops as soon as file
is present (with 0.1 sec precision). Additional sleeps shouldn't be necessary
Test Plan:
Run `test-infinitepush-*` on devserver
PYTHONPATH=/opt/facebook/hg/lib/python2.7/site-packages/:/opt/homebrew/lib/python2.7/site-packages/ /opt/homebrew/opt/python27/bin/python2.7 ../../hg-crew/tests/run-tests.py
on mac
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4381211
Signature: t1:4381211:1483551806:bbefc6632d7aed83f4a1caf8ad2ec5ba2b8f12d7
Summary:
`pushbackup` failed when there are hidden heads in the repo but no visible
heads (see test for example). This diff fixes it.
Test Plan: Run `test-infinitepush-*`
Reviewers: durham, mitrandir, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4365677
Tasks: 12479677
Signature: t1:4365677:1483465909:fd1d96211909d3eeccab4fd17a364af4a1997f85
Summary:
Recently I had this scm prompt:
{F64910275}
This is not user friendly. I suggest to output at most one remote bookmark
Also this diff copies `seq.py` from upstream hg repo
Test Plan: Run scm prompt tests on devserver and mac and with '--shell=zsh'
Reviewers: rmcelroy
Reviewed By: rmcelroy
Subscribers: simonfar, wez, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4197242
Signature: t1:4197242:1483466446:cd1fe943dc99fde0600a4e20994fa4b0f971f72a
Summary:
The '&>>' operator only works in linux, not OSX. So let's replace it with a more
verbose equivalent. 'man bash' says these two are equivalent.
Test Plan:
Running the tests normally on OSX don't actually catch this, because
this particular test is skipped because it doesn't see that evolve is installed.
We need to run the tests with:
PYTHONPATH=/opt/facebook/hg/lib/python2.7/site-packages/:/opt/homebrew/lib/python2.7/site-packages/ /opt/homebrew/opt/python27/bin/python2.7 ../../facebook-hg-rpms/hg-crew/tests/run-tests.py test-infinitepush-backup.t
To get coverage. And the test is now passing on OSX and linux.
Reviewers: stash, #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4379593
Signature: t1:4379593:1483488383:ce8a045bbc29d719eafa372db56453b83f1d8df8
Summary:
Some recent commits introduced non-OSX compatible test output. This patch fixes
that.
- 'ls' has different output text and error codes for missing files on osx
- 'sync' just waits for an in memory write to be flushed to disk, which
shouldn't actually be visible to any user land process, so it seems there's a
deeper race condition here, like wait_for_background_backup.py not working
correctly. Adding a sleep here to work around it.
Test Plan: Ran the tests on linux and osx
Reviewers: stash, #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4378618
Signature: t1:4378618:1483477739:5cd8bf82368cc417fbc7ee22410e6d962ba10342
Summary:
Without `--clean`, the user may not be able to abort cleanly.
Also, change "erase" to "discard" to be more consistent with the official
language. "changed" is replaced by "changes" as it looks like a grammar
mistake.
Test Plan: Updated existing tests
Reviewers: #sourcecontrol, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4377376
Signature: t1:4377376:1483464655:f695fd643509d7fabb88298bd1004727ee64ef76
Summary: bundle file should be closed to ensure that all the temporary files were deleted
Test Plan: arc unit
Reviewers: #sourcecontrol, ikostia
Reviewed By: ikostia
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4377148
Signature: t1:4377148:1483451990:2a1362d9e524a54ffedad8f2bc09fc622efe8ad4
Summary:
test-check-code.t was updated in core mercurial so we have to fix it in
fb-hgext
Test Plan: arc unit
Reviewers: #sourcecontrol, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4377208
Signature: t1:4377208:1483451688:bc382cdab07c6d6127d67c902458a2ccfa1373d1
Summary:
In 676596f945ea2166820ef92e692ef7fe6a6247f0 were added comments with
lines > 80. In aec81a9a80d22989bbdc8c74c1dfec9dcbbe6866 default config value
was changed.
Test Plan: arc unit
Reviewers: #sourcecontrol
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4377188
We need to keep the path in sync with the stack, and since we've already popped
the stack here, we also need to pop the path in this short-circuit case.
Previously we were relying on mfrevlog.revdiff() to produce the delta for us.
This only showed us what files were added/modified, as compared to p1, and we
had to use a heuristic to know what files were deleted (by looking at the list
of files in the commit metadata). Merge commits have a different criteria for
what is in the commit metadata (it only contains entries where the file is
different from both parents), so we can't use it for the same heuristic. So
let's fall back to a normal manifest diff for merge commits, since they are
rare.
Adds a test for verifying that conversion of merge commits into a tree works.
Summary: The --limit option was carried over from the option list for `hg log`. However, it doesn't do anything in smartlog, so this diff removes it from the option list.
Test Plan: `hg smartlog --help` no longer shows `--limit`.
Reviewers: #mercurial
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4363187
Tasks: 15186221
Summary:
Remove hard-coded devserver name and use `sync` to make sure data is written
by the background process
Test Plan: Run `test-infinitepush-*`
Reviewers: #sourcecontrol, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4361991
Tasks: 12479677
Signature: t1:4361991:1482423111:81e69044caaa608736e03cc3746cf1ff1f6625a5
Summary:
Make adding a set of rules efficient.
Currently buck autosparse writes directly to .hg/sparse, clobbering any
existing manual rules or profile settings. This is not optimal.
The command-line of hg sparse is too limited at the moment to add rules in any
other way (only one new rule per call!). The --import-rules option lets us
bulk-import and efficiently decide wether or not a refresh is needed.
Test Plan: arc unit
Reviewers: #sourcecontrol, andrasbelo
Reviewed By: andrasbelo
Subscribers: andrasbelo, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4358087
Signature: t1:4358087:1482409901:42b2d32dc624579a3d35875ba1bb126d9539cc69
Summary:
Default path was used during background backup even if non-default path was
passed on the command line. So `hg push somepath --background` is
equivalent to `hg push --background`. This is not correct and this diff fixes it
Test Plan: Run `test-infinitepush-*`
Reviewers: durham, rmcelroy, mitrandir
Reviewed By: mitrandir
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4357295
Tasks: 12479677
Signature: t1:4357295:1482330125:d2b043f8035046ab43667b4387bf642e53261681
Summary: Log active profiles for any command that uses the repository.
Test Plan: arc unit
Reviewers: #sourcecontrol, mitrandir
Reviewed By: mitrandir
Subscribers: andrasbelo, mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4346795
Signature: t1:4346795:1482251224:a9b3568eeb7e57c61273ce0dc9f4d41abcd2888e
When installed into a path like /usr/lib/python2.7/dist-packages, this file
conflicts with the hgext3rd/__init__.py from mercurial itself. Let's keep the
mercurial version the canonical one, and just skip packaging this one.
I added a warning to this repo's __init__.py about behavior divergences between
this __init__.py and mercurial's version.
Summary:
fastannotate will tell remotefilelog what nodes of a file is already known in
linelog + revmap, so remotefilelog will not prefetch those files. Previously,
fastannotate either prevents all prefetching or allows all prefetching, which
is sub-optimal.
fastannotate could now donate its sshpeer to remotefilelog, so remotefilelog
won't need to start another one (assuming they can share a same sshpeer,
could be turned off via config options). This should reduce run time if SSH
handshake is expensive.
fastannotate could now also steal sshpeer from remotefilelog, so fastannotate
won't need to start another one. Combined with the above change, there would
always be only one sshpeer shared by fastannotate and remotefilelog.
Test Plan: Modified existing tests
Reviewers: #sourcecontrol, durham
Reviewed By: durham
Subscribers: ikostia, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4325382
Signature: t1:4325382:1481933531:39d6344b2c076fbbff1f07997cd268e7cee25e80
Summary:
SCM_SAMPLING_FILEPATH is set by arcanist, and thus the tests fail when you run
arc unit.
Test Plan: Run arc unit
Reviewers: #sourcecontrol, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4342459
Signature: t1:4342459:1481929811:1ac8db74cfd86be7ad35466e6eaf2f7e35ad9140
Summary:
Logging dirstate sizes gives us a way to track sparse profile use; how big a
working copy do people actually have?
The logged info can be directed to Scuba via the sampling extension.
Test Plan: Run on a local test repo with the debugger attached. Goal
Reviewers: #sourcecontrol, quark
Reviewed By: quark
Subscribers: mitrandir, quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4340367
Signature: t1:4340367:1481920819:d7358e0bdc2cc1422b36266374344e7d2b644ae5
Summary:
Previously, if during a pushrebaes a merge commit was rebased and the
parent commit that contained the original rebase destinations content (i.e.
master's content) was on the p2 side, it would all be thrown away.
This is because the pushrebase commit works by copying the destination-repo-p1, then
applying changes from the diff of the original-repo-p1 vs original-repo-commit.
Since all the rebase destination contents was in destination-repo-p2, it is all
lost here.
The solution is to track which commit is the latest one to contain the
destination bookmark contents, and make sure it is always p1 when rebasing merge
commits. This will ensure that the destination bookmark contents are preserved
all the way up to the new commit for that bookmark.
Test Plan:
Added a test. It fails before the change and passes after. Another
test was also affected, since the p1/p2 ordering is different.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4336427
Tasks: 15092414
Signature: t1:4336427:1481881555:a22b8c363f2f7a197cef192abf34bfca926ecdc2
Summary:
Previously, the revision number was converted to binary, and parsed using the
revset language. It worked fine in some cases but could be broken when it
matches other bits of the revset language occasionally.
This patch converts the revision number to hex to address the issue.
Test Plan: `arc unit`
Reviewers: #sourcecontrol, ikostia
Reviewed By: ikostia
Subscribers: ikostia, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4327054
Signature: t1:4327054:1481904759:52170ba0ec7baf284b489f0d5a164153a7e8f25f
Summary:
When the server was handling a "getannotate" request, it may read lastnode,
update the revmap, then read lastnode again. So lastnode should not be cached.
Test Plan: `arc unit`
Reviewers: #sourcecontrol, ikostia
Reviewed By: ikostia
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4327046
Signature: t1:4327046:1481904335:2a236091b5d36872df5703e4bb9f3651012a2f02
Summary:
The default annotate command will do a "isbinary" check which reads fctx.data()
and contributes to a remotefilelog fetch round-trip. Since the "fastannotate"
command does not do binary check, and we want to minimize the round-trips,
disable the binary check by default.
Test Plan: `arc unit`
Reviewers: #sourcecontrol, ikostia
Reviewed By: ikostia
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4327028
Signature: t1:4327028:1481904237:72a59dfb622459cd961b3da615053f80b55f1a7a
Summary:
As the name suggest it will restore backup made by `hg pushbackup`.
If user has only one backup for the `dest` repo then it will be restored.
But user may have backed up many local repos that points to `dest` repo.
These local repos may reside on different hosts or in different
repo roots. It makes restore ambiguous; `--reporoot` and `--hostname`
options are used to disambiguate.
Example situation:
1) User has only one laptop with mercurial repo `repo` that was cloned
from remote server. He or she run `hg pushbackup`. Then laptop breaks
and user gets a new one, clones the `repo` again and runs `hg restore`.
It automatically restores the backup.
2) User has devserver and laptop and backups were made from both.
Then if user decides to switch devserver and run `hg restore` on the new
devserver he or she has to specify `--hostname`.
Future plans:
1) Add `--user` option to make it possible to restore another user's backup
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4280832
Tasks: 12479677
Signature: t1:4280832:1481565335:2a21ceafa2ff80242076a79693046514434afb40
Summary:
`bundlerepo` should be closed to ensure that temp file is deleted.
`bundlerepo` creates temp file when bundle is compressed. This is *not* the
case in infinitepush (all our bundle are uncompressed). But seems that
upstream cg1unpacker.compressed() returns True even if bundle is uncompressed.
Also this diff makes `_getoutputbundleraw()` read output bundle in memory
instead of returning a generator. The reason for doing it is because generator
becomes invalid as soon as `bundlerepo` is closed.
This approach will obviously increase memory usage but it shouldn't be a
problem since bundles are small (no more than 10s Mb).
#thanks @quark for reporting it
Test Plan: Run `test-infinitepush-*`
Reviewers: durham, rmcelroy, mitrandir, quark
Reviewed By: quark
Subscribers: quark, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4327186
Tasks: 12479677
Signature: t1:4327186:1481738559:2c6b22c305da4d572da9de21dfdf1179f7281744
Summary: This was added in core as well
Test Plan: tests are passing
Reviewers: #sourcecontrol, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4326595
Signature: t1:4326595:1481732429:af2ab66f73e5ef501b4d0480181715c17797b743
Summary:
Previously, when perfhack is enabled, `_matchpaths` will use slow code for
absolute paths. This diff makes it use the fast code for absolute paths.
Test Plan: `arc diff`
Reviewers: #sourcecontrol, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4318343
Signature: t1:4318343:1481724095:9d21a5135486c9d503bee3d8f93a23fa15fa647a
Summary:
The mtime check seems to cause more trouble than benefit - often the
annotate cache was treated "not up-to-date", while it is up-to-date
actually.
Test Plan: `arc unit`
Reviewers: #sourcecontrol, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4318077
Signature: t1:4318077:1481723495:d4e0ab5e7ef5a520520c07fb7246230639b9b9b4
