Summary:
Sometimes the hg process gets stuck, and we want to get the stack traces to
learn what it's doing. It's not a big issue if gdb can be used with python
debugging support [1]. However, that feature could not be easily set up on
OS X. This extension will make debugging on OS X easier by providing the
Python stack traces on SIGUSR1.
It's similar to OpenStack's "GuruMeditationReport" feature [2].
Unlike `contrib/showsstack.py`, this extension uses SIGUSR1 instead of SIGQUIT,
and writes to a file, instead of stderr, and prints all threads, instead of
just the current one. So it's more practically useful. We may want to
replace `showstack.py` eventually.
[1]: https://wiki.python.org/moin/DebuggingWithGdb
[2]: https://wiki.openstack.org/wiki/GuruMeditationReport
Test Plan: Added a new test
Reviewers: #sourcecontrol, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4660938
Signature: t1:4660938:1488927878:c751856681816a739160c361ed5cc10bab325000
Summary:
Now that ctreemanifest no longer depends on python.h, let's move pythonutil over
to cstore where all the python code is.
Test Plan: Ran the build and the tests
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4663988
Signature: t1:4663988:1488895919:652b3fc35a2dd12c51a9f70e32997c7b4d037c95
Summary:
This is the last piece of removing the Python dependency from the core
treemanifest code. This replaces the old PythonObj diff dictionary with a new
DiffResult class that has a PythonDiffResult implementation.
Test Plan:
Ran the test suite, including the unit tests that explicitly cover
treemanifest diff from python.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4663984
Signature: t1:4663984:1488889615:62f064924b0d6b45dfa7e490d19418060c374f40
Summary:
As part of breaking the native cstore implementation away from Python, let's
create a Matcher class that can be used to perform path match testing. Initially
the only implementation is the PythonMatcher which just wraps a python match
object.
Test Plan: Covered by existing matcher tests in cstore-treemanifest.py
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4663354
Signature: t1:4663354:1488914384:2c33c7e0e7f2eade0786b6ff41503317989fd1e5
Summary:
The upstream matcher has a 'visitdir' function which indicates whether a walk
should traverse into a directory. Let's support this in treemanifest so we can
skip a ton of lookups for parts of the tree that aren't relevant to the match.
Test Plan:
Used gdb to verify commands like 'hg diff -r .^ -r .
--include foo' only traversed into directory foo, and not directory bar. I'm not
sure how to write a test for this since it doesn't actually affect user facing
output.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4648252
Signature: t1:4648252:1488882860:b5e1b738fef2f63336c80cd7c0eb59c2fcf16d0f
Summary:
Previously, tree repacks did not take into account tree history. It would just
look at the delta base and if the base existed, it would just reuse the delta.
This would A) result in very long chains, and B) result in chains where the full
text was the oldest version, instead of the newest (recent full texts means
faster access to recent versions).
This patch threads tree history into the repacker, which already knows how to
use history for repacks.
Test Plan:
Updated the tests, and inspected the new test results to ensure tree
entries that were not deltas before the repack became reverse deltas during the
repack.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4647359
Signature: t1:4647359:1488882710:dba72cf488766ce827b7641735164fa0efc9a303
Summary:
Previously, when treemanifest would create packs of trees during pull, we
allowed trees to be delta'd against trees in other packs. This resulted in
smaller packs, but if the other pack disappeared for some reason (since it's a
cache), the chain broke.
This patch ensures that the first version of every tree added to a pack is a
full text.
This temporarily makes repacks worse, since the repacker doesn't know about
history to produce deltas when combining packs. The next patch adds history
awareness which improves the repack deltafication.
Test Plan:
Updated the tests, and inspected the new test results to ensure that
all packs only had deltas within the pack.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4647348
Signature: t1:4647348:1488882214:e850622a853a534fc60caeef604c88c30740c60d
Summary:
Previously the treemanifest code itself would create the text deltas when
writing a tree out. This meant we couldn't make the delta decision based on
other data, like if the p1 commit was in the same pack file.
This patch removes treemanifest.write() and moves all calls over to
treemanifest.finalize() which gives the python/pack layer control over delta
choices. A future patch will use this to ensure tree packs always contain
complete delta chains.
Test Plan: All tests pass
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4645942
Signature: t1:4645942:1488880851:d0c8c902e7e849072a53344630a9184b6d8e1e7f
Summary:
Previously, to serialize a tree we would call treemanifest.write() which would
take the stores and call the appropriate add() functions. This meant the
treemanifest code controlled when deltas were used, which makes it hard to
decide deltas based on external factors, like which pack file the parent is in.
As a first step to fixing this, we're adding a new finalize function that
returns a python iterator that allows iterating over the new parts of the tree.
In a future diff we will use this to do serialization and delta decisions at the
python/pack layer instead.
The same future diff will also cover this in tests, as it moves all calls to
treemanifest.write() to use treemanifest.finalize().
Test Plan: The next diff switches all write() calls and tests to finalize()
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4645922
Signature: t1:4645922:1488880722:574a37ba800f4b78faf8bad3979b447f4278cc55
Summary:
Previously the treemanifest auto-tree-creation logic only produced data packs
containing the actual contents of the tree blobs. This lost history information
which is important for our ability to efficiently repack the data files.
This patch creates history packs during pull as well. A future patch will also
create history packs for the local tree blob store.
Test Plan: Updated the tests to cover this
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4638865
Signature: t1:4638865:1488449992:48b60961b50b90b6d0e75a64af1f36fb29944e7a
Summary:
To support treemanifests in history packs we need to support the empty filename
(i.e. the root of the repo). This removes some checks that prevented that from
working.
Test Plan:
A future patch will add history pack support for treemanifests,
including tests that cover this.
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4637917
Signature: t1:4637917:1488429178:c25d03b73eb379d4126ebbcee4bb5797f7b841b2
Summary:
Upstream changed the api of this function. Let's add args and kwargs to cover
future api changes.
Test Plan: Ran the tests
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4667282
Signature: t1:4667282:1488910879:7db5764704dd345946f7f933e7e2c4a8fd77b699
Summary:
Upstream has deprecated repo.*opener properties, so we need to switch to the
equivalent vfs apis to prevent us from being spammed with deprecation warnings.
Test Plan: Ran the tests
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4667252
Signature: t1:4667252:1488911019:efc3adf21f723151f1ee3b21b91fe62ea3c7b259
Summary: manifestctx's upstream now accept a manifestlog instead of a repo.
Test Plan: Ran the tests
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4667246
Signature: t1:4667246:1488911047:5340f4dc18183c68dfd47a0c5223b0f804e17806
Summary:
Command that checks that every head and bookmark node is present in the
bundlestore
Test Plan: arc unit
Reviewers: #mercurial
Subscribers: #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4658436
Tasks: 15389402
Summary: This functionality will be used in command to check backup consistency
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4658433
Tasks: 15389402
Signature: t1:4658433:1488828061:49aa5bfeba922d617cf1aab393f5e9f598aa33a4
Summary:
Computing cache key can be quite slow (100ms and even more). Instead let's use
a simpler cache key: size + mtime of phaseroots, obsstore and changelod and
whether changelog write was delayed.
Test Plan:
1) arc unit
2) Run test-newhiddencache.t
3) Run upstream unittest with extension enabled.
python run-tests.py --extra-config-opt extensions.newhiddencachekey=/home/stash/facebook-hg-rpms/fb-hgext/hgext3rd/newhiddencachekey.py -j20
Verify that there are no failures except for those that show that new extension were added.
Note: also need to comment out debug output in the extension
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4604811
Signature: t1:4604811:1487958882:702ea8fdee16def674db97344832936d53f0cd9d
Summary:
This small change makes a way better user experience. Users usually have one
big repo per host, but they may have many hosts (for example, laptop and
devserver). If we ask for reporoot first then it's very likely that next
`pullbackup` invocation will fail because there will be ambigious hosts.
Let's switch the order.
Test Plan: arc unit
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4649946
Tasks: 15389402
Signature: t1:4649946:1488561645:330590fcf1dcd4af7fb572c5d4ccfd8a5ab78c60
Summary:
Next diff will add special option to specify whose backup to restore.
This diff does a necessary refactoring
Test Plan: arc unit
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4650282
Tasks: 15389402
Signature: t1:4650282:1488562203:632a8c84d6d537663bd7d94ce9dfd18a4498ccfb
Summary:
This diff is part of the series to avoid downloading the same bundle
a few times.
Finally reuse the same bundlerepo.
Test Plan: arc unit
Reviewers: #mercurial
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4612094
Tasks: 15389402
Summary:
This diff is part of the series to avoid downloading the same bundle
a few times.
Now close all bundlerepos at once not one by one. This is necessary because
bundlerepos will be reused.
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4612089
Tasks: 15389402
Signature: t1:4612089:1488501776:e9fc2863adebc69bf44fa5dcb1610612027508f6
Summary:
This diff is part of the series to avoid downloading the same bundle
a few times.
Test Plan: arc unti
Reviewers: #mercurial, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4612073
Tasks: 15389402
Signature: t1:4612073:1487955965:03af1a25acc21d0b6d913a60dd5ef82fa7f033ad
Summary:
We added a use of std::unordered_set::emplace, which isn't available in
GCC 4.4 (thus CentOS 6). Let's try insert instead, as that's the only other way
to add to an unordered_set.
Test Plan: Build on CentOS 7 to confirm no regression.
Reviewers: #sourcecontrol
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4642804
Summary:
There are a couple of reasons to avoid using ui.username():
1) If config option ui.username is set on the server then it will use it
instead of the name of the user that does push
2) It prints confusing warning `no username found, using '....' instead`
3) In some cases it fails (probably because it calls socket.getfqdn() which
does network request).
Let's use simple util.getuser() function and set username to 'unknown' if
it fails.
Test Plan: arc unit
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: azich, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4642457
Signature: t1:4642457:1488458040:5dfac435bf44dfa6ab7725d2f5800963e8c860ec
Summary:
Upstream mercurial doesn't run hooks for temporary amended commits
(see https://bz.mercurial-scm.org/show_bug.cgi?id=3501 and
c2ca20984e3c9f30d73f0f35c35904b64edbd692). No other hook contains info about
temporary amended commits.
Let's wrap localrepository.commit() function and add missing commits inside
this wrapper.
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4634950
Tasks: 16387175
Signature: t1:4634950:1488399457:1d299c6ad3cfd418dda32a7ec20c60290f8d7778
Summary:
It makes it easier to set up a hook - just enabling the extension will enable
the hook.
Test Plan:
Run perl script
system("echo 1 >> 1");
system("hg add 1");
system("hg ci -m"."\x80");
system("hg ci -m"."\x01");
system("hg ci -m ok");
make sure only one commit was created
Reviewers: #sourcecontrol, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4604980
Tasks: 16212973
Signature: t1:4604980:1487944300:8e4e7f340c746237b47773fc81759025e97b919e
Summary:
bundlerepo adds new commits that are not present in fastpartialmatch index.
Without this diff any access to the commit from bundle fails if
fastpartialmatch.raiseifinconsistent config option is set.
Let's not use fastpartialmatch in bundlerepos
Test Plan: arc unit
Reviewers: #mercurial, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4635368
Tasks: 16394593
Signature: t1:4635368:1488393506:15b8f6a9fddabe681de7fd1296bbfafb9512f558
Summary:
Turned out that wrapping `changectx.__init__()` doesn't give any performance
benefits but instead it breaks initializing of changectx. For example, if
there is a commit which revision number N is a prefix of another commit A then
`hg log -r N` outputs commit A instead of commit N. Let's get rid of this
wrapping.
Test Plan: arc unit
Reviewers: #mercurial, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4635311
Tasks: 16394593
Signature: t1:4635311:1488393427:eae091a0d155df336adbb122f2881a28e95ba6b6
Summary:
Adds a treemanifest.usecunionstore config flag for enabling and disabling use of
the native code uniondatapackstore.
Since we haven't implemented the repack APIs on the native datapack stores, we
currently have to force repack to use the old python implementations. Instead of
trying to expose just the appropriate APIs through the python interface, I think
we'll rewrite all of repack to be in C++ at a future time, since we can take
advantage of parallelism, etc.
Test Plan:
Updated test-treemanifest.t to use the c datapackstore. Also run all
the tests with --extra-config-opt=treemanifest.usecunionstore=True.
These tests caught a missing null check in the C++ code as well.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4609795
Signature: t1:4609795:1488365341:203362db5f470b613c4d6484686cd32c3fa8458f
Summary:
The markForRefresh function allows callers to tell the datapackstore that new
pack files have been written. This is important for being able to support
generation of treemanifests during pull and is a blocker for using the native
datapackstore in production.
Test Plan: Added a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4609076
Signature: t1:4609076:1488365022:d5480897f2b7674b4d07243e20a495551f985321
Summary: Make `test-show.t` pass again!
Test Plan: - run test locally, see it passing
Reviewers: #sourcecontrol, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4636012
Signature: t1:4636012:1488401012:da051bd55c91890c637f954aaf151d88ea37a978
Summary: As per upstream's https://www.mercurial-scm.org/repo/hg-committed/rev/8fb2c95d816a, we need to add `--color` to help messages.
Test Plan: - run test locally, see it passing
Reviewers: #sourcecontrol, mitrandir, simonfar
Reviewed By: mitrandir, simonfar
Subscribers: simonfar, mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4627818
Signature: t1:4627818:1488296052:eff62a80e71c24965072d13e2f67b0e78b3f4ea3
Summary:
This makes the `_getfiles` batch size configurable. Let me know if some other config name will serve this purpose better.
**Reason for this fix**
Currently, `_getfile` will write 10000 lines of text into a pipe and only upon the success of this operation, will read file blobs from another pipe. Serving process will start writing file blobs into a second pipe as soon as it sees something in the first pipe. Second pipe's buffer will fill up as it is not read from by the client until client writes 10K file requests. 10K file requests fill the buffer of the first pipe and we have a deadlock.
Ideally, we should make client check whether it can write to the first pipe and if not, go and read from the second pipe, but that is a bigger fix.
Test Plan:
- run local tests, see them all passing
- except for `test-cstore.t`, but it fails for me without my changes as well
- this generally makes sense
Reviewers: #sourcecontrol, durham
Reviewed By: durham
Subscribers: durham, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4620152
Signature: t1:4620152:1488221258:04555177926d129c6ba41bc982ad4e913cb31b20
Summary:
Core mercurial relies on readfast being fast for both iteration and containment
checks. Our previous implementation just returned the full manifest, which was
fast for containment (when a tree was available), but very slow for iteration.
The changegroup code iterates over the result of readfast in an attempt to
record linkrev information as revisions are applied. This resulted in very slow
bundle application in some cases since each full iteration of a treemanifest can
take several seconds.
This fix makes us use tree diffs when possible, which make that iteration very
fast again.
Test Plan:
I had a repro for a slow hg bundle command in our large repo. Before,
each full iteration over the readfast results took 11+ seconds, and now it takes
effectively 0 seconds. The resulting bundles were the same (technically the new
bundle was smaller, since the smarter change detection meant less data was added
to the bundle).
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters, mbolin
Differential Revision: https://phabricator.intern.facebook.com/D4619223
Signature: t1:4619223:1488162752:0a39d7488209cdf74af6ebb6ab27eb1b5873c76a
Summary:
The sparse profile in .hg/sparse currently only can grow, removing rules from this must either be done manually (by editing the file), or one by one using the the -d/--delete command line switch.
To mirror the hg sparse --import-rules command line, hg sparse --clear-rules removes local includes/excludes rules.
Test Plan: fb-hgext/tests/test-sparse-clear.t
Reviewers: marianomartin, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4609102
Tasks: 16073155
Signature: t1:4609102:1488028035:f5f7b7420d911667fb4a07ac770b7e0480a76977
Summary:
We have users who don't use inhibit, so printing this message clutters their UI.
Let's just move this debug.
Test Plan: yolo
Reviewers: kulshrax, #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4614036
Signature: t1:4614036:1487986024:6945b057aae62fc7a9fded9474d47ba134496e5a
Summary:
The current code checks if the filter error reports a rev number, and if it does
tries to convert it to a node. If the 'number' is actually just a short form of
a hash that happens to be all digits, then it may be a number greater than the
size of the changelog, so we need to not attempt to access it in the changelog,
otherwise we get an IndexError.
Test Plan: Added a test, it failed before with a weird error
Reviewers: #mercurial, kulshrax
Reviewed By: kulshrax
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4613988
Signature: t1:4613988:1488027938:4f5ffcb07aeecbaf59cad20fb3a592dd92ed2beb
Summary: Get rid of hardcoded devserver name
Test Plan: arc unit
Reviewers: #sourcecontrol, andrasbelo
Reviewed By: andrasbelo
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4611903
Signature: t1:4611903:1487951306:00465fe52de92c71f85c3706594d1a128d4b9628
Summary:
Previously --list-remote wasn't user friendly. User have to always specify a
pattern, and specifying * just won't work because of bash globbing. Also
it wasn't possible to specify many patterns at once.
This diff allows specifying many patterns, it allows to not specify
patterns at all and also it allows to specify default scratch patterns to fetch
if no patterns were specified. This is useful if there are many auto-generated
bookmarks under the same scratch bookmark pattern.
Test Plan: arc unit
Reviewers: #mercurial, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4605548
Signature: t1:4605548:1487934838:066a12c28dda16fa8f90674d736d21345631ef7e
