Summary:
Computing cache key can be quite slow (100ms and even more). Instead let's use
a simpler cache key: size + mtime of phaseroots, obsstore and changelod and
whether changelog write was delayed.
Test Plan:
1) arc unit
2) Run test-newhiddencache.t
3) Run upstream unittest with extension enabled.
python run-tests.py --extra-config-opt extensions.newhiddencachekey=/home/stash/facebook-hg-rpms/fb-hgext/hgext3rd/newhiddencachekey.py -j20
Verify that there are no failures except for those that show that new extension were added.
Note: also need to comment out debug output in the extension
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4604811
Signature: t1:4604811:1487958882:702ea8fdee16def674db97344832936d53f0cd9d
Summary:
This small change makes a way better user experience. Users usually have one
big repo per host, but they may have many hosts (for example, laptop and
devserver). If we ask for reporoot first then it's very likely that next
`pullbackup` invocation will fail because there will be ambigious hosts.
Let's switch the order.
Test Plan: arc unit
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4649946
Tasks: 15389402
Signature: t1:4649946:1488561645:330590fcf1dcd4af7fb572c5d4ccfd8a5ab78c60
Summary:
Next diff will add special option to specify whose backup to restore.
This diff does a necessary refactoring
Test Plan: arc unit
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4650282
Tasks: 15389402
Signature: t1:4650282:1488562203:632a8c84d6d537663bd7d94ce9dfd18a4498ccfb
Summary:
This diff is part of the series to avoid downloading the same bundle
a few times.
Finally reuse the same bundlerepo.
Test Plan: arc unit
Reviewers: #mercurial
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4612094
Tasks: 15389402
Summary:
This diff is part of the series to avoid downloading the same bundle
a few times.
Now close all bundlerepos at once not one by one. This is necessary because
bundlerepos will be reused.
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4612089
Tasks: 15389402
Signature: t1:4612089:1488501776:e9fc2863adebc69bf44fa5dcb1610612027508f6
Summary:
This diff is part of the series to avoid downloading the same bundle
a few times.
Test Plan: arc unti
Reviewers: #mercurial, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4612073
Tasks: 15389402
Signature: t1:4612073:1487955965:03af1a25acc21d0b6d913a60dd5ef82fa7f033ad
Summary:
We added a use of std::unordered_set::emplace, which isn't available in
GCC 4.4 (thus CentOS 6). Let's try insert instead, as that's the only other way
to add to an unordered_set.
Test Plan: Build on CentOS 7 to confirm no regression.
Reviewers: #sourcecontrol
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4642804
Summary:
There are a couple of reasons to avoid using ui.username():
1) If config option ui.username is set on the server then it will use it
instead of the name of the user that does push
2) It prints confusing warning `no username found, using '....' instead`
3) In some cases it fails (probably because it calls socket.getfqdn() which
does network request).
Let's use simple util.getuser() function and set username to 'unknown' if
it fails.
Test Plan: arc unit
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: azich, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4642457
Signature: t1:4642457:1488458040:5dfac435bf44dfa6ab7725d2f5800963e8c860ec
Summary:
Upstream mercurial doesn't run hooks for temporary amended commits
(see https://bz.mercurial-scm.org/show_bug.cgi?id=3501 and
c2ca20984e3c9f30d73f0f35c35904b64edbd692). No other hook contains info about
temporary amended commits.
Let's wrap localrepository.commit() function and add missing commits inside
this wrapper.
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4634950
Tasks: 16387175
Signature: t1:4634950:1488399457:1d299c6ad3cfd418dda32a7ec20c60290f8d7778
Summary:
It makes it easier to set up a hook - just enabling the extension will enable
the hook.
Test Plan:
Run perl script
system("echo 1 >> 1");
system("hg add 1");
system("hg ci -m"."\x80");
system("hg ci -m"."\x01");
system("hg ci -m ok");
make sure only one commit was created
Reviewers: #sourcecontrol, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4604980
Tasks: 16212973
Signature: t1:4604980:1487944300:8e4e7f340c746237b47773fc81759025e97b919e
Summary:
bundlerepo adds new commits that are not present in fastpartialmatch index.
Without this diff any access to the commit from bundle fails if
fastpartialmatch.raiseifinconsistent config option is set.
Let's not use fastpartialmatch in bundlerepos
Test Plan: arc unit
Reviewers: #mercurial, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4635368
Tasks: 16394593
Signature: t1:4635368:1488393506:15b8f6a9fddabe681de7fd1296bbfafb9512f558
Summary:
Turned out that wrapping `changectx.__init__()` doesn't give any performance
benefits but instead it breaks initializing of changectx. For example, if
there is a commit which revision number N is a prefix of another commit A then
`hg log -r N` outputs commit A instead of commit N. Let's get rid of this
wrapping.
Test Plan: arc unit
Reviewers: #mercurial, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4635311
Tasks: 16394593
Signature: t1:4635311:1488393427:eae091a0d155df336adbb122f2881a28e95ba6b6
Summary:
Adds a treemanifest.usecunionstore config flag for enabling and disabling use of
the native code uniondatapackstore.
Since we haven't implemented the repack APIs on the native datapack stores, we
currently have to force repack to use the old python implementations. Instead of
trying to expose just the appropriate APIs through the python interface, I think
we'll rewrite all of repack to be in C++ at a future time, since we can take
advantage of parallelism, etc.
Test Plan:
Updated test-treemanifest.t to use the c datapackstore. Also run all
the tests with --extra-config-opt=treemanifest.usecunionstore=True.
These tests caught a missing null check in the C++ code as well.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4609795
Signature: t1:4609795:1488365341:203362db5f470b613c4d6484686cd32c3fa8458f
Summary:
The markForRefresh function allows callers to tell the datapackstore that new
pack files have been written. This is important for being able to support
generation of treemanifests during pull and is a blocker for using the native
datapackstore in production.
Test Plan: Added a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4609076
Signature: t1:4609076:1488365022:d5480897f2b7674b4d07243e20a495551f985321
Summary: Make `test-show.t` pass again!
Test Plan: - run test locally, see it passing
Reviewers: #sourcecontrol, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4636012
Signature: t1:4636012:1488401012:da051bd55c91890c637f954aaf151d88ea37a978
Summary: As per upstream's https://www.mercurial-scm.org/repo/hg-committed/rev/8fb2c95d816a, we need to add `--color` to help messages.
Test Plan: - run test locally, see it passing
Reviewers: #sourcecontrol, mitrandir, simonfar
Reviewed By: mitrandir, simonfar
Subscribers: simonfar, mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4627818
Signature: t1:4627818:1488296052:eff62a80e71c24965072d13e2f67b0e78b3f4ea3
Summary:
This makes the `_getfiles` batch size configurable. Let me know if some other config name will serve this purpose better.
**Reason for this fix**
Currently, `_getfile` will write 10000 lines of text into a pipe and only upon the success of this operation, will read file blobs from another pipe. Serving process will start writing file blobs into a second pipe as soon as it sees something in the first pipe. Second pipe's buffer will fill up as it is not read from by the client until client writes 10K file requests. 10K file requests fill the buffer of the first pipe and we have a deadlock.
Ideally, we should make client check whether it can write to the first pipe and if not, go and read from the second pipe, but that is a bigger fix.
Test Plan:
- run local tests, see them all passing
- except for `test-cstore.t`, but it fails for me without my changes as well
- this generally makes sense
Reviewers: #sourcecontrol, durham
Reviewed By: durham
Subscribers: durham, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4620152
Signature: t1:4620152:1488221258:04555177926d129c6ba41bc982ad4e913cb31b20
Summary:
Core mercurial relies on readfast being fast for both iteration and containment
checks. Our previous implementation just returned the full manifest, which was
fast for containment (when a tree was available), but very slow for iteration.
The changegroup code iterates over the result of readfast in an attempt to
record linkrev information as revisions are applied. This resulted in very slow
bundle application in some cases since each full iteration of a treemanifest can
take several seconds.
This fix makes us use tree diffs when possible, which make that iteration very
fast again.
Test Plan:
I had a repro for a slow hg bundle command in our large repo. Before,
each full iteration over the readfast results took 11+ seconds, and now it takes
effectively 0 seconds. The resulting bundles were the same (technically the new
bundle was smaller, since the smarter change detection meant less data was added
to the bundle).
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters, mbolin
Differential Revision: https://phabricator.intern.facebook.com/D4619223
Signature: t1:4619223:1488162752:0a39d7488209cdf74af6ebb6ab27eb1b5873c76a
Summary:
The sparse profile in .hg/sparse currently only can grow, removing rules from this must either be done manually (by editing the file), or one by one using the the -d/--delete command line switch.
To mirror the hg sparse --import-rules command line, hg sparse --clear-rules removes local includes/excludes rules.
Test Plan: fb-hgext/tests/test-sparse-clear.t
Reviewers: marianomartin, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4609102
Tasks: 16073155
Signature: t1:4609102:1488028035:f5f7b7420d911667fb4a07ac770b7e0480a76977
Summary:
We have users who don't use inhibit, so printing this message clutters their UI.
Let's just move this debug.
Test Plan: yolo
Reviewers: kulshrax, #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4614036
Signature: t1:4614036:1487986024:6945b057aae62fc7a9fded9474d47ba134496e5a
Summary:
The current code checks if the filter error reports a rev number, and if it does
tries to convert it to a node. If the 'number' is actually just a short form of
a hash that happens to be all digits, then it may be a number greater than the
size of the changelog, so we need to not attempt to access it in the changelog,
otherwise we get an IndexError.
Test Plan: Added a test, it failed before with a weird error
Reviewers: #mercurial, kulshrax
Reviewed By: kulshrax
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4613988
Signature: t1:4613988:1488027938:4f5ffcb07aeecbaf59cad20fb3a592dd92ed2beb
Summary: Get rid of hardcoded devserver name
Test Plan: arc unit
Reviewers: #sourcecontrol, andrasbelo
Reviewed By: andrasbelo
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4611903
Signature: t1:4611903:1487951306:00465fe52de92c71f85c3706594d1a128d4b9628
Summary:
Previously --list-remote wasn't user friendly. User have to always specify a
pattern, and specifying * just won't work because of bash globbing. Also
it wasn't possible to specify many patterns at once.
This diff allows specifying many patterns, it allows to not specify
patterns at all and also it allows to specify default scratch patterns to fetch
if no patterns were specified. This is useful if there are many auto-generated
bookmarks under the same scratch bookmark pattern.
Test Plan: arc unit
Reviewers: #mercurial, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4605548
Signature: t1:4605548:1487934838:066a12c28dda16fa8f90674d736d21345631ef7e
Summary:
discovery.findcommonoutgoing() explicitly skips secret commits. Because of it
we also need to skip bookmarks that point to secret commits, otherwise they
will point to non-existent nodes.
Test Plan: arc unit
Reviewers: #mercurial, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4611393
Tasks: 15389402
Signature: t1:4611393:1487933210:72e79923c944b13204f4cde64d415076703bbe47
Summary:
Previously treemanifest always called back to Python to fetch the tree content
for a given key. This patch let's us call directly in to the cstore without
going through Python, when possible.
Test Plan:
I manually tested this by modifying the treemanifest extension to
pass a uniondatapackstore to the ctreemanifest constructor, then used the
debugger to ensure this code path was correct. I still need to add a test
though.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4571442
Signature: t1:4571442:1487847314:0d578351354ee4da34b5032c01720bdedf0d185d
Summary:
In a future patch we will want to pass the uniondatapackstore around to other
objects who will contribute to the lifetime. Let's change it to a shared_ptr so
that becomes easy.
Let's also make the destructor virtual, so we can pass different types of stores
around and have them be destructed correctly.
Test Plan: Ran the tests
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4603893
Signature: t1:4603893:1487847173:2fc3505032ea8c30cf9e0f76ac4e75d64513d87d
Summary:
A future patch will want to import just the cstore py headers into
py-ctreemanifest, so let's move those out to their own header file.
Test Plan: Ran the tests
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4571320
Signature: t1:4571320:1487774077:64b4c5053b223f075a1c9f73ddcd92bd73b53e23
Summary:
The old code kept a PythonObj around inside the ManifestFetcher for fetching
manifest contents from the store. As part of moving the treemanifest code to use
the new native cstore API let's make the manifest code depend on a Store
abstraction and have one implementation be a PythonStore.
This removes almost all of the python dependencies from the core treemanifest
code, except some logic around running the python matcher during iteration and
writing directly to the python result dict during diff. We'll abstract those
away later.
Test Plan: Built and ran the tests
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4569944
Signature: t1:4569944:1487847102:d005b6484fd7de9335961b0bc4530505b25f961d
Summary:
To begin the process of weaning treemanifest off of Python and on to the new
cstore, let's introduce a base Store class that implements a simple get()
function. UnionDatapackStore already implements this. Future patches will make a
Python version that implements this interface as well, so we can swap between
them easily.
Test Plan: Built and ran the tests.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4569870
Signature: t1:4569870:1487846232:aa816311ec37a520a2405b68e3c02ad744613049
Summary:
This implements the get function of UnionDatapackStore which iterates over
deltas and produces a full text. Unfortunately the delta chains currently come
out in reverse order (full text last), so we have to read all of them before we
can produce the delta.
Since mpatch takes a char*, and we want to avoid copying the array into a string
array for every read, we event a ref counted string reference type which let's
us transfer ownership of the char* to the caller.
Test Plan: Added a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: stash, simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556547
Signature: t1:4556547:1487200302:c73b8f464f6ce413f1c22c0125a91fbece701f08
Summary:
A future patch will add C++ logic that applies delta's to get full texts, so we
need access to the mpatch code. This is a verbatim copy from core, along with
it's dependencies.
Test Plan: N/A It gets used as part of the next patch
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556537
Signature: t1:4556537:1487076858:528343cb0a74de9262bbb5927ec8d186dafaef45
Summary:
Implements the getdeltachain function on the new UnionDatapackStore class.
This required some modifications to the DeltaChainIterator. Since the results of
the iterator may cross multiple different chains, we need to keep each chain
alive until the iterator is destructed, so we need to keep a reference to each
chain. We also had to remove the size() property from the iterator since the
fact that the chain spans chains means we don't know the size up front.
Test Plan: Adds a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556458
Signature: t1:4556458:1487199872:07dffa3121acfbeb6d6993b518e6f4887122d4d5
Summary:
This adds a new C++ UnionDatapackStore implementation that only has the
getmissing() function at the moment.
Test Plan: Adds a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556052
Signature: t1:4556052:1487161607:664752df19c63c06819ee2af5b4c436f1b76609d
Summary:
This implements a C++ version of the datapackstore getmissing() function and
adds a test.
Test Plan: Added a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, ikostia, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4552853
Signature: t1:4552853:1487191581:c108be4a85f8eacafddbea8086d5d22fdcabdacf
Summary:
Building in debug mode was failing because python emits some warning about some
define needing to only be used in opt builds. Since we have -Werror, this gets
treated like an error. Let's not use -Werror in debug builds.
Test Plan: hg purge --all && FB_HGEXT_CDEBUG=1 python setup.py build --component cstore
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4552872
Signature: t1:4552872:1487075732:35b639bc0abd0b2d70b8e0dedd83f35c26396b10
Summary:
The remaining python parts of the store are a perf bottleneck when accessing
hundreds of thousands of pack file entries (like in treemanifest). Let's
implement them in C++.
This first patch just add the basic boiler plate, and implements a single
function getdeltachain(), with a test. Future patches will add more
functionality and other parts of the store.
Since cstore depends on cdatapack and ctreemanifest (the pythonutils.h part for
now), we need to tweak our setup.py to enforce a certain build order too.
Test Plan: Added a test, yo
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4547929
Signature: t1:4547929:1487181318:21c146cf370d26cb97efe6a883868b85b4e32f49
Summary:
As part of unifying our native store data structures into a single library,
let's move the treemanifest (including the python extension) into py-cstore.
Test Plan:
Built and ran the tests. Verified there was no ctreemanifest.so
dependency in the built cstore.so by using 'ldd cstore.so' on Linux and 'otools
-L cstore.so' on OSX.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4602484
Signature: t1:4602484:1487842683:964cbb43b7cb20d0db699ef691fe7fcf6bccf2e8
Summary:
'python setup.py build' puts the build artifacts in build/lib.../, but
test-ctreemanifest.py was not configured to look there so it was actually
running against the system ctreemanifest in that case.
The old development workflow was to use make local, which put it at the root of
the repo, which is why this used to work. Let's move the ctreemanifest to be
with the rest of the tests in test-cstore.t so we can set the environment up
correctly.
Test Plan:
Some future diffs broke the test without me knowing. I happened to
catch it via other means, so I came back and fixed the test framework.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4570449
Signature: t1:4570449:1487763880:682800284c01ac25bd99cbe811a8d98d34caf79b
Summary:
As part of unifying our storage layer into a single library, let's move
py-cdatapack into the new cstore directory. Future patches will move
ctreemanifest and the upcoming datapackstore into here as well.
py-cdatapack.h required some reordering since it seems forward declarations work
a little differently between C and C++. There were no code changes though,
except one int->size_t fix.
Test Plan: Ran the tests
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4581320
Signature: t1:4581320:1487788968:e8a34c7a03a16db282214c7dd476b749b92a1bfa
Summary:
A future patch which renamed cdatapack.so to cstore.so pointed out that our
test-remotefilelog*.py tests are running against the system remotefilelog. Let's
move them to a .t test that sets the appropriate PYTHONPATH.
Test Plan: Ran the tests
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4581281
Signature: t1:4581281:1487788843:365c6adf41ba51bdbe7394c1ab423fcc5a63bdbc
