facebook/sapling
1
1
Fork 0
mirror of https://github.com/facebook/sapling.git synced 2024-10-09 16:31:02 +03:00
Code Issues Projects Releases Wiki Activity
a6782ea273
sapling/tests/test-fb-hgext-fastpartialmatch.t

471 lines
14 KiB
Perl
Raw Normal View History

fastpartialmatch: initial commit Summary: Index to make node partial match fast i. e. speed up matching of partial hashes like f9bbd5. This is an initial diff and it doesn't add any useful functionality, just debug commands. Storage format is simple. There are a few files (at most 256). Each file contains entries: <20-byte node hash><4 byte encoded rev> Each entry represents a commit (node). Name of the file is the first two letters of the hex node hash. Nodes with the same first two letters go to the same file. Nodes are NOT sorted inside the file to make appends of new nodes easier. Partial index should always be correct i.e. it should contain only nodes that are present in the repo (regardless of whether they are visible or not) and rev numbers for nodes should be correct too. Test Plan: arc unit Reviewers: #sourcecontrol Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4474642
2017-02-02 16:12:10 +03:00
$ mkcommit() {
> echo "$1" > "$1"
> hg add "$1"
> hg ci -m "$1"
> }
$ cat >> $HGRCPATH << EOF
> [extensions]
fb-hgext: fix copied fb-hgext tests Summary: This is a big bulk of generally almost-obvious fixes to the moved tests. Mostly these fixes have to do with correct importing of the actual extensions. Depends on D6675329 Test Plan: - ./run-tests.py fails less after this commit - see further commits for more test fixes Reviewers: #sourcecontrol Differential Revision: https://phabricator.intern.facebook.com/D6675344
2018-01-09 14:06:09 +03:00
> fastpartialmatch=
fastpartialmatch: initial commit Summary: Index to make node partial match fast i. e. speed up matching of partial hashes like f9bbd5. This is an initial diff and it doesn't add any useful functionality, just debug commands. Storage format is simple. There are a few files (at most 256). Each file contains entries: <20-byte node hash><4 byte encoded rev> Each entry represents a commit (node). Name of the file is the first two letters of the hex node hash. Nodes with the same first two letters go to the same file. Nodes are NOT sorted inside the file to make appends of new nodes easier. Partial index should always be correct i.e. it should contain only nodes that are present in the repo (regardless of whether they are visible or not) and rev numbers for nodes should be correct too. Test Plan: arc unit Reviewers: #sourcecontrol Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4474642
2017-02-02 16:12:10 +03:00
> strip=
> histedit=
> [ui]
> ssh = python "$TESTDIR/dummyssh"
> EOF
$ hg init repo
$ cd repo
fastpartialmatch: add commit hook Summary: Add commit hook to append new entries to the index. It uses the same transactions as revlogs. Do not append entries if index is not built (i.e. if _partialindex does not exist). This is to ensure that we either have complete index or none at all. Besides next diff will add hooks to build index after pull if it doesn't exist. Test Plan: arc unit Reviewers: #sourcecontrol Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4474654
2017-02-02 16:12:10 +03:00
$ hg debugrebuildpartialindex
fastpartialmatch: initial commit Summary: Index to make node partial match fast i. e. speed up matching of partial hashes like f9bbd5. This is an initial diff and it doesn't add any useful functionality, just debug commands. Storage format is simple. There are a few files (at most 256). Each file contains entries: <20-byte node hash><4 byte encoded rev> Each entry represents a commit (node). Name of the file is the first two letters of the hex node hash. Nodes with the same first two letters go to the same file. Nodes are NOT sorted inside the file to make appends of new nodes easier. Partial index should always be correct i.e. it should contain only nodes that are present in the repo (regardless of whether they are visible or not) and rev numbers for nodes should be correct too. Test Plan: arc unit Reviewers: #sourcecontrol Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4474642
2017-02-02 16:12:10 +03:00
$ mkcommit "first"
fastpartialmatch: add commit hook Summary: Add commit hook to append new entries to the index. It uses the same transactions as revlogs. Do not append entries if index is not built (i.e. if _partialindex does not exist). This is to ensure that we either have complete index or none at all. Besides next diff will add hooks to build index after pull if it doesn't exist. Test Plan: arc unit Reviewers: #sourcecontrol Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4474654
2017-02-02 16:12:10 +03:00
$ hg debugcheckpartialindex
fastpartialmatch: initial commit Summary: Index to make node partial match fast i. e. speed up matching of partial hashes like f9bbd5. This is an initial diff and it doesn't add any useful functionality, just debug commands. Storage format is simple. There are a few files (at most 256). Each file contains entries: <20-byte node hash><4 byte encoded rev> Each entry represents a commit (node). Name of the file is the first two letters of the hex node hash. Nodes with the same first two letters go to the same file. Nodes are NOT sorted inside the file to make appends of new nodes easier. Partial index should always be correct i.e. it should contain only nodes that are present in the repo (regardless of whether they are visible or not) and rev numbers for nodes should be correct too. Test Plan: arc unit Reviewers: #sourcecontrol Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4474642
2017-02-02 16:12:10 +03:00
$ hg log -r . -T '{node}\n'
b75a450e74d5a7708da8c3144fbeb4ac88694044
fastpartialmatch: sort the nodes during rebuilding Summary: We'll add bisect algorithm soon. Let's sort the nodes first during index rebuilding Test Plan: arc unit Reviewers: #sourcecontrol Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4488263
2017-02-02 16:12:10 +03:00
Check permissions
fastpartialmatch: add generation number Summary: fastpartialmatch index may be incorrect if fastpartialmatch extension was disabled, then new commits were added and then fastpartialmatch was enabled back. We need to detect this situation and rebuild the index in this case. One approach would be to use _handleinconsistentindex() function to mark index as invalid. There are a couple of drawbacks to this approach: 1) It requires more significant code changes 2) The output may still be incorrect if hash with the same prefix was added while fastpartialmatch was disabled. Another approach would be to use header version from index files (see class _header). That's even worse because we'll need to bump index version in the code whenever we disable fastpartialmatch. Third approach would be to use changelog mtime + size as a validation key but then we need to ensure partialmatch index is written after changelog. Instead I suggest to use generation numbers. It's just a number in the config and in the special file inside partialmatch index directory. If the two doesn't match then index is incorrect and needs to be deleted. The biggest drawback to this approach is that we need to bump generation number each time the extension is enabled. I think it should be fine because the one who enables extensions usually understands it and he or she will bump the generation number. Test Plan: arc unit Reviewers: #sourcecontrol, durham, mjpieters Reviewed By: mjpieters Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4571926 Signature: t1:4571926:1487263035:53419c9ca4e344127f391ac19c1071a826fea3f1
2017-02-16 21:18:22 +03:00
$ ls -al .hg/store/partialindex/ | sort
-rw-r--r--.* generationnum (re)
-rw-r--r--.* b7 (re)
fastpartialmatch: fix tests Test Plan: arc unit Reviewers: #sourcecontrol, andrasbelo Reviewed By: andrasbelo Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4507168 Signature: t1:4507168:1486110887:01175dc859d68a6668cb4d51cc7b343db843084f
2017-02-03 11:37:10 +03:00
drwxr-xr-x.* \. (re)
drwxr-xr-x.* \.\. (re)
fastpartialmatch: add generation number Summary: fastpartialmatch index may be incorrect if fastpartialmatch extension was disabled, then new commits were added and then fastpartialmatch was enabled back. We need to detect this situation and rebuild the index in this case. One approach would be to use _handleinconsistentindex() function to mark index as invalid. There are a couple of drawbacks to this approach: 1) It requires more significant code changes 2) The output may still be incorrect if hash with the same prefix was added while fastpartialmatch was disabled. Another approach would be to use header version from index files (see class _header). That's even worse because we'll need to bump index version in the code whenever we disable fastpartialmatch. Third approach would be to use changelog mtime + size as a validation key but then we need to ensure partialmatch index is written after changelog. Instead I suggest to use generation numbers. It's just a number in the config and in the special file inside partialmatch index directory. If the two doesn't match then index is incorrect and needs to be deleted. The biggest drawback to this approach is that we need to bump generation number each time the extension is enabled. I think it should be fine because the one who enables extensions usually understands it and he or she will bump the generation number. Test Plan: arc unit Reviewers: #sourcecontrol, durham, mjpieters Reviewed By: mjpieters Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4571926 Signature: t1:4571926:1487263035:53419c9ca4e344127f391ac19c1071a826fea3f1
2017-02-16 21:18:22 +03:00
total \d+ (re)
fastpartialmatch: sort the nodes during rebuilding Summary: We'll add bisect algorithm soon. Let's sort the nodes first during index rebuilding Test Plan: arc unit Reviewers: #sourcecontrol Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4488263
2017-02-02 16:12:10 +03:00
fastpartialmatch: initial commit Summary: Index to make node partial match fast i. e. speed up matching of partial hashes like f9bbd5. This is an initial diff and it doesn't add any useful functionality, just debug commands. Storage format is simple. There are a few files (at most 256). Each file contains entries: <20-byte node hash><4 byte encoded rev> Each entry represents a commit (node). Name of the file is the first two letters of the hex node hash. Nodes with the same first two letters go to the same file. Nodes are NOT sorted inside the file to make appends of new nodes easier. Partial index should always be correct i.e. it should contain only nodes that are present in the repo (regardless of whether they are visible or not) and rev numbers for nodes should be correct too. Test Plan: arc unit Reviewers: #sourcecontrol Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4474642
2017-02-02 16:12:10 +03:00
Check debug commands
$ hg debugrebuildpartialindex
fastpartialmatch: add commit hook Summary: Add commit hook to append new entries to the index. It uses the same transactions as revlogs. Do not append entries if index is not built (i.e. if _partialindex does not exist). This is to ensure that we either have complete index or none at all. Besides next diff will add hooks to build index after pull if it doesn't exist. Test Plan: arc unit Reviewers: #sourcecontrol Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4474654
2017-02-02 16:12:10 +03:00
$ hg debugcheckpartialindex
fastpartialmatch: initial commit Summary: Index to make node partial match fast i. e. speed up matching of partial hashes like f9bbd5. This is an initial diff and it doesn't add any useful functionality, just debug commands. Storage format is simple. There are a few files (at most 256). Each file contains entries: <20-byte node hash><4 byte encoded rev> Each entry represents a commit (node). Name of the file is the first two letters of the hex node hash. Nodes with the same first two letters go to the same file. Nodes are NOT sorted inside the file to make appends of new nodes easier. Partial index should always be correct i.e. it should contain only nodes that are present in the repo (regardless of whether they are visible or not) and rev numbers for nodes should be correct too. Test Plan: arc unit Reviewers: #sourcecontrol Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4474642
2017-02-02 16:12:10 +03:00
$ hg debugprintpartialindexfile
abort: please specify a filename
[255]
$ hg debugprintpartialindexfile unknownfile
file unknownfile does not exist
$ hg debugprintpartialindexfile b7
b75a450e74d5a7708da8c3144fbeb4ac88694044 0
Check that debugcheckpartialindex fails on corrupted indexes
$ hg debugcheckpartialindex
$ rm .hg/store/partialindex/b7
$ hg debugcheckpartialindex
b75a450e74d5a7708da8c3144fbeb4ac88694044 node not found in partialindex
[1]
$ printf 'garbage' > .hg/store/partialindex/b7
$ hg debugcheckpartialindex
fastpartialmatch: add header Summary: Append header to each fast partial match index file. For now it contains version number and number of sorted entries in the file. File is ignored if version number from file is different. Test Plan: arc unit Reviewers: #sourcecontrol Differential Revision: https://phabricator.intern.facebook.com/D4488417
2017-02-02 16:12:10 +03:00
b7 file is corrupted: corrupted header: run `hg debugrebuildpartialindex` to fix the issue
fastpartialmatch: initial commit Summary: Index to make node partial match fast i. e. speed up matching of partial hashes like f9bbd5. This is an initial diff and it doesn't add any useful functionality, just debug commands. Storage format is simple. There are a few files (at most 256). Each file contains entries: <20-byte node hash><4 byte encoded rev> Each entry represents a commit (node). Name of the file is the first two letters of the hex node hash. Nodes with the same first two letters go to the same file. Nodes are NOT sorted inside the file to make appends of new nodes easier. Partial index should always be correct i.e. it should contain only nodes that are present in the repo (regardless of whether they are visible or not) and rev numbers for nodes should be correct too. Test Plan: arc unit Reviewers: #sourcecontrol Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4474642
2017-02-02 16:12:10 +03:00
b75a450e74d5a7708da8c3144fbeb4ac88694044 node not found in partialindex
[1]
fastpartialmatch: wrap revlog._partialmatch and changectx.__init__ Summary: These two functions actually use fastpartial index to resolve revision for a node. Test Plan: arc unit Reviewers: #sourcecontrol Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4474658
2017-02-02 16:12:10 +03:00
$ hg log -r b75a
fastpartialmatch: add header Summary: Append header to each fast partial match index file. For now it contains version number and number of sorted entries in the file. File is ignored if version number from file is different. Test Plan: arc unit Reviewers: #sourcecontrol Differential Revision: https://phabricator.intern.facebook.com/D4488417
2017-02-02 16:12:10 +03:00
failed to read partial index partialindex/b7 : corrupted header: run `hg debugrebuildpartialindex` to fix the issue
failed to read partial index partialindex/b7 : corrupted header: run `hg debugrebuildpartialindex` to fix the issue
fastpartialmatch: wrap revlog._partialmatch and changectx.__init__ Summary: These two functions actually use fastpartial index to resolve revision for a node. Test Plan: arc unit Reviewers: #sourcecontrol Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4474658
2017-02-02 16:12:10 +03:00
changeset: 0:b75a450e74d5
tag: tip
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: first
fastpartialmatch: initial commit Summary: Index to make node partial match fast i. e. speed up matching of partial hashes like f9bbd5. This is an initial diff and it doesn't add any useful functionality, just debug commands. Storage format is simple. There are a few files (at most 256). Each file contains entries: <20-byte node hash><4 byte encoded rev> Each entry represents a commit (node). Name of the file is the first two letters of the hex node hash. Nodes with the same first two letters go to the same file. Nodes are NOT sorted inside the file to make appends of new nodes easier. Partial index should always be correct i.e. it should contain only nodes that are present in the repo (regardless of whether they are visible or not) and rev numbers for nodes should be correct too. Test Plan: arc unit Reviewers: #sourcecontrol Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4474642
2017-02-02 16:12:10 +03:00
$ mkcommit committostrip
$ hg log -r . -T '{node}'
1138fa1e0b22411fc96c825c2603c5c3d056a206 (no-eol)
$ hg debugrebuildpartialindex
$ mv .hg/store/partialindex .hg/store/tmppartialindex
$ hg strip .
0 files updated, 0 files merged, 1 files removed, 0 files unresolved
saved backup bundle to $TESTTMP/repo/.hg/strip-backup/1138fa1e0b22-27b827b8-backup.hg (glob)
$ mv .hg/store/tmppartialindex .hg/store/partialindex
$ hg debugcheckpartialindex
abort: 00changelog.i@1138fa1e0b22: no node!
[255]
$ hg debugrebuildpartialindex
$ hg debugcheckpartialindex
fastpartialmatch: wrap revlog._partialmatch and changectx.__init__ Summary: These two functions actually use fastpartial index to resolve revision for a node. Test Plan: arc unit Reviewers: #sourcecontrol Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4474658
2017-02-02 16:12:10 +03:00
Resolve 0 revision. Make sure index is not used
$ hg log -r 0 --debug | egrep 'changeset|using partial index cache'
changeset: 0:b75a450e74d5a7708da8c3144fbeb4ac88694044
Resolve by commit hash prefix. Make sure index is used
$ hg log -r b75a --debug | egrep 'changeset|using partial index cache'
using partial index cache 0
using partial index cache 0
changeset: 0:b75a450e74d5a7708da8c3144fbeb4ac88694044
Try to resolve unknown hash
$ hg log -r ololo
abort: unknown revision 'ololo'!
[255]
Test raiseifinconsistent option
$ rm .hg/store/partialindex/b7
$ hg log -r b75a
changeset: 0:b75a450e74d5
tag: tip
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: first
$ hg log --config fastpartialmatch.raiseifinconsistent=True -r b75a 2>&1 | grep ValueError
ValueError: inconsistent partial match index while resolving b75a
fastpartialmatch: pretxnchangegroup hook Summary: This hook builds index if it's not present and it appends new entries from the changegroup. Test Plan: arc unit Reviewers: #sourcecontrol Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4474704
2017-02-02 16:12:10 +03:00
Test that new partial index entries are created during clone and pull
$ cd ..
$ hg clone -q ssh://user@dummy/repo cloned
$ cd cloned
fastpartialmatch: add generation number Summary: fastpartialmatch index may be incorrect if fastpartialmatch extension was disabled, then new commits were added and then fastpartialmatch was enabled back. We need to detect this situation and rebuild the index in this case. One approach would be to use _handleinconsistentindex() function to mark index as invalid. There are a couple of drawbacks to this approach: 1) It requires more significant code changes 2) The output may still be incorrect if hash with the same prefix was added while fastpartialmatch was disabled. Another approach would be to use header version from index files (see class _header). That's even worse because we'll need to bump index version in the code whenever we disable fastpartialmatch. Third approach would be to use changelog mtime + size as a validation key but then we need to ensure partialmatch index is written after changelog. Instead I suggest to use generation numbers. It's just a number in the config and in the special file inside partialmatch index directory. If the two doesn't match then index is incorrect and needs to be deleted. The biggest drawback to this approach is that we need to bump generation number each time the extension is enabled. I think it should be fine because the one who enables extensions usually understands it and he or she will bump the generation number. Test Plan: arc unit Reviewers: #sourcecontrol, durham, mjpieters Reviewed By: mjpieters Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4571926 Signature: t1:4571926:1487263035:53419c9ca4e344127f391ac19c1071a826fea3f1
2017-02-16 21:18:22 +03:00
$ ls .hg/store/partialindex | sort
fastpartialmatch: pretxnchangegroup hook Summary: This hook builds index if it's not present and it appends new entries from the changegroup. Test Plan: arc unit Reviewers: #sourcecontrol Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4474704
2017-02-02 16:12:10 +03:00
b7
fastpartialmatch: add generation number Summary: fastpartialmatch index may be incorrect if fastpartialmatch extension was disabled, then new commits were added and then fastpartialmatch was enabled back. We need to detect this situation and rebuild the index in this case. One approach would be to use _handleinconsistentindex() function to mark index as invalid. There are a couple of drawbacks to this approach: 1) It requires more significant code changes 2) The output may still be incorrect if hash with the same prefix was added while fastpartialmatch was disabled. Another approach would be to use header version from index files (see class _header). That's even worse because we'll need to bump index version in the code whenever we disable fastpartialmatch. Third approach would be to use changelog mtime + size as a validation key but then we need to ensure partialmatch index is written after changelog. Instead I suggest to use generation numbers. It's just a number in the config and in the special file inside partialmatch index directory. If the two doesn't match then index is incorrect and needs to be deleted. The biggest drawback to this approach is that we need to bump generation number each time the extension is enabled. I think it should be fine because the one who enables extensions usually understands it and he or she will bump the generation number. Test Plan: arc unit Reviewers: #sourcecontrol, durham, mjpieters Reviewed By: mjpieters Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4571926 Signature: t1:4571926:1487263035:53419c9ca4e344127f391ac19c1071a826fea3f1
2017-02-16 21:18:22 +03:00
generationnum
fastpartialmatch: pretxnchangegroup hook Summary: This hook builds index if it's not present and it appends new entries from the changegroup. Test Plan: arc unit Reviewers: #sourcecontrol Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4474704
2017-02-02 16:12:10 +03:00
$ cd ../repo
$ mkcommit fromserver
$ hg log -r . -T '{node}\n'
3dd368d533d16f6172e27321f05f9a419ca354bf
$ cd ../cloned
$ hg pull -q
fastpartialmatch: add generation number Summary: fastpartialmatch index may be incorrect if fastpartialmatch extension was disabled, then new commits were added and then fastpartialmatch was enabled back. We need to detect this situation and rebuild the index in this case. One approach would be to use _handleinconsistentindex() function to mark index as invalid. There are a couple of drawbacks to this approach: 1) It requires more significant code changes 2) The output may still be incorrect if hash with the same prefix was added while fastpartialmatch was disabled. Another approach would be to use header version from index files (see class _header). That's even worse because we'll need to bump index version in the code whenever we disable fastpartialmatch. Third approach would be to use changelog mtime + size as a validation key but then we need to ensure partialmatch index is written after changelog. Instead I suggest to use generation numbers. It's just a number in the config and in the special file inside partialmatch index directory. If the two doesn't match then index is incorrect and needs to be deleted. The biggest drawback to this approach is that we need to bump generation number each time the extension is enabled. I think it should be fine because the one who enables extensions usually understands it and he or she will bump the generation number. Test Plan: arc unit Reviewers: #sourcecontrol, durham, mjpieters Reviewed By: mjpieters Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4571926 Signature: t1:4571926:1487263035:53419c9ca4e344127f391ac19c1071a826fea3f1
2017-02-16 21:18:22 +03:00
$ ls .hg/store/partialindex | sort
fastpartialmatch: pretxnchangegroup hook Summary: This hook builds index if it's not present and it appends new entries from the changegroup. Test Plan: arc unit Reviewers: #sourcecontrol Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4474704
2017-02-02 16:12:10 +03:00
3d
b7
fastpartialmatch: add generation number Summary: fastpartialmatch index may be incorrect if fastpartialmatch extension was disabled, then new commits were added and then fastpartialmatch was enabled back. We need to detect this situation and rebuild the index in this case. One approach would be to use _handleinconsistentindex() function to mark index as invalid. There are a couple of drawbacks to this approach: 1) It requires more significant code changes 2) The output may still be incorrect if hash with the same prefix was added while fastpartialmatch was disabled. Another approach would be to use header version from index files (see class _header). That's even worse because we'll need to bump index version in the code whenever we disable fastpartialmatch. Third approach would be to use changelog mtime + size as a validation key but then we need to ensure partialmatch index is written after changelog. Instead I suggest to use generation numbers. It's just a number in the config and in the special file inside partialmatch index directory. If the two doesn't match then index is incorrect and needs to be deleted. The biggest drawback to this approach is that we need to bump generation number each time the extension is enabled. I think it should be fine because the one who enables extensions usually understands it and he or she will bump the generation number. Test Plan: arc unit Reviewers: #sourcecontrol, durham, mjpieters Reviewed By: mjpieters Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4571926 Signature: t1:4571926:1487263035:53419c9ca4e344127f391ac19c1071a826fea3f1
2017-02-16 21:18:22 +03:00
generationnum
fastpartialmatch: pretxnchangegroup hook Summary: This hook builds index if it's not present and it appends new entries from the changegroup. Test Plan: arc unit Reviewers: #sourcecontrol Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4474704
2017-02-02 16:12:10 +03:00
$ hg debugprintpartialindexfile 3d
3dd368d533d16f6172e27321f05f9a419ca354bf 1
Remove partial index and make sure everything still works
$ rm -r .hg/store/partialindex
$ hg log -r 3dd368
changeset: 1:3dd368d533d1
tag: tip
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: fromserver
$ hg debugrebuildpartialindex
fastpartialmatch: handle strips correctly Summary: strip may change revision numbers of many nodes at once. It's better to rebuild index in case of strip. Test Plan: arc unit Reviewers: #sourcecontrol Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4474710
2017-02-02 16:12:10 +03:00
Test strip
$ hg log --graph
o changeset: 1:3dd368d533d1
| tag: tip
| user: test
| date: Thu Jan 01 00:00:00 1970 +0000
| summary: fromserver
|
@ changeset: 0:b75a450e74d5
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: first
$ hg strip -r 1
saved backup bundle to $TESTTMP/cloned/.hg/strip-backup/3dd368d533d1-aec0bb31-backup.hg (glob)
$ hg log --graph
@ changeset: 0:b75a450e74d5
tag: tip
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: first
$ hg debugcheckpartialindex
Try to commit after partial index was stripped
$ mkcommit afterstrip
$ hg debugcheckpartialindex
Test histedit
$ mkcommit tohistedit
$ hg log --graph
@ changeset: 2:353c4093de9e
| tag: tip
| user: test
| date: Thu Jan 01 00:00:00 1970 +0000
| summary: tohistedit
|
o changeset: 1:51e0111a3ca1
| user: test
| date: Thu Jan 01 00:00:00 1970 +0000
| summary: afterstrip
|
o changeset: 0:b75a450e74d5
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: first
$ hg histedit --commands - <<EOF
> pick 353c4093de9e 2 tohistedit
> pick 51e0111a3ca1 1 afterstrip
> EOF
tests: update to match upstream backup file name change Summary: Upstream made these changes. Let's follow along. Test Plan: Ran the tests Reviewers: #fbhgext, quark Reviewed By: #fbhgext, quark Differential Revision: https://phab.mercurial-scm.org/D46
2017-07-11 23:19:40 +03:00
saved backup bundle to $TESTTMP/cloned/.hg/strip-backup/51e0111a3ca1-ae8f0808-histedit.hg (glob)
fastpartialmatch: handle strips correctly Summary: strip may change revision numbers of many nodes at once. It's better to rebuild index in case of strip. Test Plan: arc unit Reviewers: #sourcecontrol Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4474710
2017-02-02 16:12:10 +03:00
$ hg log --graph
@ changeset: 2:8dc08dfc2ed8
| tag: tip
| user: test
| date: Thu Jan 01 00:00:00 1970 +0000
| summary: afterstrip
|
o changeset: 1:1d6b19400cfd
| user: test
| date: Thu Jan 01 00:00:00 1970 +0000
| summary: tohistedit
|
o changeset: 0:b75a450e74d5
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: first
fastpartialmatch: add generation number Summary: fastpartialmatch index may be incorrect if fastpartialmatch extension was disabled, then new commits were added and then fastpartialmatch was enabled back. We need to detect this situation and rebuild the index in this case. One approach would be to use _handleinconsistentindex() function to mark index as invalid. There are a couple of drawbacks to this approach: 1) It requires more significant code changes 2) The output may still be incorrect if hash with the same prefix was added while fastpartialmatch was disabled. Another approach would be to use header version from index files (see class _header). That's even worse because we'll need to bump index version in the code whenever we disable fastpartialmatch. Third approach would be to use changelog mtime + size as a validation key but then we need to ensure partialmatch index is written after changelog. Instead I suggest to use generation numbers. It's just a number in the config and in the special file inside partialmatch index directory. If the two doesn't match then index is incorrect and needs to be deleted. The biggest drawback to this approach is that we need to bump generation number each time the extension is enabled. I think it should be fine because the one who enables extensions usually understands it and he or she will bump the generation number. Test Plan: arc unit Reviewers: #sourcecontrol, durham, mjpieters Reviewed By: mjpieters Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4571926 Signature: t1:4571926:1487263035:53419c9ca4e344127f391ac19c1071a826fea3f1
2017-02-16 21:18:22 +03:00
$ ls .hg/store/partialindex | sort
fastpartialmatch: handle strips correctly Summary: strip may change revision numbers of many nodes at once. It's better to rebuild index in case of strip. Test Plan: arc unit Reviewers: #sourcecontrol Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4474710
2017-02-02 16:12:10 +03:00
1d
8d
b7
fastpartialmatch: add generation number Summary: fastpartialmatch index may be incorrect if fastpartialmatch extension was disabled, then new commits were added and then fastpartialmatch was enabled back. We need to detect this situation and rebuild the index in this case. One approach would be to use _handleinconsistentindex() function to mark index as invalid. There are a couple of drawbacks to this approach: 1) It requires more significant code changes 2) The output may still be incorrect if hash with the same prefix was added while fastpartialmatch was disabled. Another approach would be to use header version from index files (see class _header). That's even worse because we'll need to bump index version in the code whenever we disable fastpartialmatch. Third approach would be to use changelog mtime + size as a validation key but then we need to ensure partialmatch index is written after changelog. Instead I suggest to use generation numbers. It's just a number in the config and in the special file inside partialmatch index directory. If the two doesn't match then index is incorrect and needs to be deleted. The biggest drawback to this approach is that we need to bump generation number each time the extension is enabled. I think it should be fine because the one who enables extensions usually understands it and he or she will bump the generation number. Test Plan: arc unit Reviewers: #sourcecontrol, durham, mjpieters Reviewed By: mjpieters Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4571926 Signature: t1:4571926:1487263035:53419c9ca4e344127f391ac19c1071a826fea3f1
2017-02-16 21:18:22 +03:00
generationnum
fastpartialmatch: handle strips correctly Summary: strip may change revision numbers of many nodes at once. It's better to rebuild index in case of strip. Test Plan: arc unit Reviewers: #sourcecontrol Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4474710
2017-02-02 16:12:10 +03:00
Abort strip and then restore. Check partial index
$ rm -rf .hg/strip-backup/*
$ printf '\n[hooks]\npriority.pretxnclose.fastpartialmatch=10' >> .hg/hgrc
$ hg strip -r . --config hooks.pretxnclose.abort=false
0 files updated, 0 files merged, 1 files removed, 0 files unresolved
saved backup bundle to $TESTTMP/cloned/.hg/strip-backup/8dc08dfc2ed8-5905654b-backup.hg (glob)
transaction abort!
rollback completed
strip failed, backup bundle stored in '$TESTTMP/cloned/.hg/strip-backup/8dc08dfc2ed8-5905654b-backup.hg'
abort: pretxnclose.abort hook exited with status 1
[255]
$ hg log --graph
@ changeset: 1:1d6b19400cfd
| tag: tip
| user: test
| date: Thu Jan 01 00:00:00 1970 +0000
| summary: tohistedit
|
o changeset: 0:b75a450e74d5
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: first
$ hg debugcheckpartialindex
$ hg unbundle -q .hg/strip-backup/*
$ hg log --graph
o changeset: 2:8dc08dfc2ed8
| tag: tip
| user: test
| date: Thu Jan 01 00:00:00 1970 +0000
| summary: afterstrip
|
@ changeset: 1:1d6b19400cfd
| user: test
| date: Thu Jan 01 00:00:00 1970 +0000
| summary: tohistedit
|
o changeset: 0:b75a450e74d5
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: first
$ hg debugcheckpartialindex
Abort a commit
$ echo 1 > 1
$ hg add 1
$ hg commit -m 'commit to discard' --config hooks.pretxnclose.abort=false
transaction abort!
rollback completed
abort: pretxnclose.abort hook exited with status 1
[255]
$ hg forget 1
$ rm 1
$ hg debugcheckpartialindex
Make clone with fastpartialmatch disabled. Make pull, make sure partial index
is rebuilt
$ cd ..
$ hg clone --config extensions.fastpartialmatch=! -q ssh://user@dummy/repo cloned2
$ cd repo
$ mkcommit newcommit
$ cd ../cloned2
$ hg pull -q
$ hg log --graph -T '{node}'
o 64905857be49e6c1134f69f9743ecf8ac04a93e2
|
@ 3dd368d533d16f6172e27321f05f9a419ca354bf
|
o b75a450e74d5a7708da8c3144fbeb4ac88694044
$ hg debugcheckpartialindex
Now crash during pull
$ cd ..
$ rm -rf cloned2
$ hg clone -q ssh://user@dummy/repo cloned2
$ cd repo
$ mkcommit secondcrashpull
$ hg log -r . -T '{node}\n'
ac536ed8bde0682e30bb64c64570758903ce1aa6
$ cd ../cloned2
$ hg debugcheckpartialindex
$ hg pull -q --config hooks.pretxnclose.abort=false
transaction abort!
rollback completed
abort: pretxnclose.abort hook exited with status 1
[255]
$ hg debugprintpartialindexfile ac
$ hg pull -q
$ hg debugprintpartialindexfile ac
ac536ed8bde0682e30bb64c64570758903ce1aa6 3
fastpartialmatch: use bisect to quickly find matching revisions Summary: Since we have some of the nodes sorted we can now use bisect to quickly find matching nodes. Test Plan: arc unit Reviewers: #sourcecontrol Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4494419
2017-02-02 16:12:10 +03:00
$ hg log --graph -T '{node} {rev}'
o ac536ed8bde0682e30bb64c64570758903ce1aa6 3
|
@ 64905857be49e6c1134f69f9743ecf8ac04a93e2 2
|
o 3dd368d533d16f6172e27321f05f9a419ca354bf 1
|
o b75a450e74d5a7708da8c3144fbeb4ac88694044 0
$ hg debugresolvepartialhash ac536e
ac536e: ac536ed8bde0682e30bb64c64570758903ce1aa6 3
Test usebisect option
$ hg debugrebuildpartialindex
$ hg --config fastpartialmatch.usebisect=False debugresolvepartialhash ac536e
ac536e: ac536ed8bde0682e30bb64c64570758903ce1aa6 3
fastpartialmatch: trigger rebuilds if there were too many unsorted entries Summary: If we have too many unsorted entries then finding matching revisions is slower. To fix it we will rebuild partial index from time to time. _findcandidates function marks index as need-to-be-rebuilt if it notices index file with too many unsorted entries. Then _changegrouphook rebuilds the index. Test Plan: arc unit Reviewers: #sourcecontrol Differential Revision: https://phabricator.intern.facebook.com/D4494617
2017-02-02 16:12:10 +03:00
Test cache rebuilding
$ cd ../repo
$ mkcommit committotriggercacherebuilding
$ cd ../cloned2
$ printf '[fastpartialmatch]\nunsortedthreshold=1' >> .hg/hgrc
$ hg up -q ac536ed8bde0
fastpartialmatch: fix devel.warn Summary: Upstream hg recently introduced new developer warning if files inside .hg are modified without taking a lock. The usual fix is to add file in localrepository._wlockfreeprefix list, but it doesn't work for "needrebuild" fastpartialmatch file because it's stored inside .hg/store. So to get rid of developer warning let's move needrebuild file to .hg/. And since we are detecting that partial index needs to be rebuilt in revlog class, I also had to wrap `runcommand()` and create needrebuild file there, because `runcommand()` has vfs object, while revlog class doesn't. Test Plan: Run tests Reviewers: #fbhgext, durham Reviewed By: #fbhgext, durham Subscribers: durham Differential Revision: https://phab.mercurial-scm.org/D99
2017-07-17 14:03:15 +03:00
$ [ -f .hg/partialindexneedrebuild ]
[1]
fastpartialmatch: trigger rebuilds if there were too many unsorted entries Summary: If we have too many unsorted entries then finding matching revisions is slower. To fix it we will rebuild partial index from time to time. _findcandidates function marks index as need-to-be-rebuilt if it notices index file with too many unsorted entries. Then _changegrouphook rebuilds the index. Test Plan: arc unit Reviewers: #sourcecontrol Differential Revision: https://phabricator.intern.facebook.com/D4494617
2017-02-02 16:12:10 +03:00
$ mkcommit commit
$ hg log -r . -T '{node}\n'
587cd78c6d0eb0259484b09a5983bcc2973f2245
Next command should set that cache needs rebuilding
$ hg log -r 587cd78c6d0eb0259 > /dev/null
fastpartialmatch: fix devel.warn Summary: Upstream hg recently introduced new developer warning if files inside .hg are modified without taking a lock. The usual fix is to add file in localrepository._wlockfreeprefix list, but it doesn't work for "needrebuild" fastpartialmatch file because it's stored inside .hg/store. So to get rid of developer warning let's move needrebuild file to .hg/. And since we are detecting that partial index needs to be rebuilt in revlog class, I also had to wrap `runcommand()` and create needrebuild file there, because `runcommand()` has vfs object, while revlog class doesn't. Test Plan: Run tests Reviewers: #fbhgext, durham Reviewed By: #fbhgext, durham Subscribers: durham Differential Revision: https://phab.mercurial-scm.org/D99
2017-07-17 14:03:15 +03:00
$ [ -f .hg/partialindexneedrebuild ]
fastpartialmatch: trigger rebuilds if there were too many unsorted entries Summary: If we have too many unsorted entries then finding matching revisions is slower. To fix it we will rebuild partial index from time to time. _findcandidates function marks index as need-to-be-rebuilt if it notices index file with too many unsorted entries. Then _changegrouphook rebuilds the index. Test Plan: arc unit Reviewers: #sourcecontrol Differential Revision: https://phabricator.intern.facebook.com/D4494617
2017-02-02 16:12:10 +03:00
$ hg debugfastpartialmatchstat
fastpartialmatch: add generation number Summary: fastpartialmatch index may be incorrect if fastpartialmatch extension was disabled, then new commits were added and then fastpartialmatch was enabled back. We need to detect this situation and rebuild the index in this case. One approach would be to use _handleinconsistentindex() function to mark index as invalid. There are a couple of drawbacks to this approach: 1) It requires more significant code changes 2) The output may still be incorrect if hash with the same prefix was added while fastpartialmatch was disabled. Another approach would be to use header version from index files (see class _header). That's even worse because we'll need to bump index version in the code whenever we disable fastpartialmatch. Third approach would be to use changelog mtime + size as a validation key but then we need to ensure partialmatch index is written after changelog. Instead I suggest to use generation numbers. It's just a number in the config and in the special file inside partialmatch index directory. If the two doesn't match then index is incorrect and needs to be deleted. The biggest drawback to this approach is that we need to bump generation number each time the extension is enabled. I think it should be fine because the one who enables extensions usually understands it and he or she will bump the generation number. Test Plan: arc unit Reviewers: #sourcecontrol, durham, mjpieters Reviewed By: mjpieters Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4571926 Signature: t1:4571926:1487263035:53419c9ca4e344127f391ac19c1071a826fea3f1
2017-02-16 21:18:22 +03:00
generation number: 0
fastpartialmatch: trigger rebuilds if there were too many unsorted entries Summary: If we have too many unsorted entries then finding matching revisions is slower. To fix it we will rebuild partial index from time to time. _findcandidates function marks index as need-to-be-rebuilt if it notices index file with too many unsorted entries. Then _changegrouphook rebuilds the index. Test Plan: arc unit Reviewers: #sourcecontrol Differential Revision: https://phabricator.intern.facebook.com/D4494617
2017-02-02 16:12:10 +03:00
index will be rebuilt on the next pull
file: 3d, entries: 1, out of them 1 sorted
fastpartialmatch: fix the build Summary: Sort the output to fix tests on different platforms Test Plan: arc unit on mac Reviewers: #sourcecontrol, simonfar, ikostia Reviewed By: simonfar, ikostia Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4501449 Signature: t1:4501449:1486050577:3041dc3c3691f48e39a0e9e05b1f97954a6d88e2
2017-02-02 19:00:55 +03:00
file: 58, entries: 1, out of them 0 sorted
file: 64, entries: 1, out of them 1 sorted
fastpartialmatch: trigger rebuilds if there were too many unsorted entries Summary: If we have too many unsorted entries then finding matching revisions is slower. To fix it we will rebuild partial index from time to time. _findcandidates function marks index as need-to-be-rebuilt if it notices index file with too many unsorted entries. Then _changegrouphook rebuilds the index. Test Plan: arc unit Reviewers: #sourcecontrol Differential Revision: https://phabricator.intern.facebook.com/D4494617
2017-02-02 16:12:10 +03:00
file: ac, entries: 1, out of them 1 sorted
file: b7, entries: 1, out of them 1 sorted
Now do a pull and make sure that index was rebuilt (file '12' is not rebuilt
because it was just pulled)
$ hg pull -q
$ hg debugfastpartialmatchstat
fastpartialmatch: add generation number Summary: fastpartialmatch index may be incorrect if fastpartialmatch extension was disabled, then new commits were added and then fastpartialmatch was enabled back. We need to detect this situation and rebuild the index in this case. One approach would be to use _handleinconsistentindex() function to mark index as invalid. There are a couple of drawbacks to this approach: 1) It requires more significant code changes 2) The output may still be incorrect if hash with the same prefix was added while fastpartialmatch was disabled. Another approach would be to use header version from index files (see class _header). That's even worse because we'll need to bump index version in the code whenever we disable fastpartialmatch. Third approach would be to use changelog mtime + size as a validation key but then we need to ensure partialmatch index is written after changelog. Instead I suggest to use generation numbers. It's just a number in the config and in the special file inside partialmatch index directory. If the two doesn't match then index is incorrect and needs to be deleted. The biggest drawback to this approach is that we need to bump generation number each time the extension is enabled. I think it should be fine because the one who enables extensions usually understands it and he or she will bump the generation number. Test Plan: arc unit Reviewers: #sourcecontrol, durham, mjpieters Reviewed By: mjpieters Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4571926 Signature: t1:4571926:1487263035:53419c9ca4e344127f391ac19c1071a826fea3f1
2017-02-16 21:18:22 +03:00
generation number: 0
fastpartialmatch: fix the build Summary: Sort the output to fix tests on different platforms Test Plan: arc unit on mac Reviewers: #sourcecontrol, simonfar, ikostia Reviewed By: simonfar, ikostia Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4501449 Signature: t1:4501449:1486050577:3041dc3c3691f48e39a0e9e05b1f97954a6d88e2
2017-02-02 19:00:55 +03:00
file: 12, entries: 1, out of them 0 sorted
fastpartialmatch: trigger rebuilds if there were too many unsorted entries Summary: If we have too many unsorted entries then finding matching revisions is slower. To fix it we will rebuild partial index from time to time. _findcandidates function marks index as need-to-be-rebuilt if it notices index file with too many unsorted entries. Then _changegrouphook rebuilds the index. Test Plan: arc unit Reviewers: #sourcecontrol Differential Revision: https://phabricator.intern.facebook.com/D4494617
2017-02-02 16:12:10 +03:00
file: 3d, entries: 1, out of them 1 sorted
fastpartialmatch: fix the build Summary: Sort the output to fix tests on different platforms Test Plan: arc unit on mac Reviewers: #sourcecontrol, simonfar, ikostia Reviewed By: simonfar, ikostia Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4501449 Signature: t1:4501449:1486050577:3041dc3c3691f48e39a0e9e05b1f97954a6d88e2
2017-02-02 19:00:55 +03:00
file: 58, entries: 1, out of them 1 sorted
file: 64, entries: 1, out of them 1 sorted
fastpartialmatch: trigger rebuilds if there were too many unsorted entries Summary: If we have too many unsorted entries then finding matching revisions is slower. To fix it we will rebuild partial index from time to time. _findcandidates function marks index as need-to-be-rebuilt if it notices index file with too many unsorted entries. Then _changegrouphook rebuilds the index. Test Plan: arc unit Reviewers: #sourcecontrol Differential Revision: https://phabricator.intern.facebook.com/D4494617
2017-02-02 16:12:10 +03:00
file: ac, entries: 1, out of them 1 sorted
file: b7, entries: 1, out of them 1 sorted
Increase unsortedthreshold and make one more pull. Make sure index doesn't need
to be rebuilt
$ cd ../repo
$ mkcommit somecommit
$ cd ../cloned2
$ hg pull -q --config fastpartialmatch.unsortedthreshold=2
$ hg debugfastpartialmatchstat
fastpartialmatch: add generation number Summary: fastpartialmatch index may be incorrect if fastpartialmatch extension was disabled, then new commits were added and then fastpartialmatch was enabled back. We need to detect this situation and rebuild the index in this case. One approach would be to use _handleinconsistentindex() function to mark index as invalid. There are a couple of drawbacks to this approach: 1) It requires more significant code changes 2) The output may still be incorrect if hash with the same prefix was added while fastpartialmatch was disabled. Another approach would be to use header version from index files (see class _header). That's even worse because we'll need to bump index version in the code whenever we disable fastpartialmatch. Third approach would be to use changelog mtime + size as a validation key but then we need to ensure partialmatch index is written after changelog. Instead I suggest to use generation numbers. It's just a number in the config and in the special file inside partialmatch index directory. If the two doesn't match then index is incorrect and needs to be deleted. The biggest drawback to this approach is that we need to bump generation number each time the extension is enabled. I think it should be fine because the one who enables extensions usually understands it and he or she will bump the generation number. Test Plan: arc unit Reviewers: #sourcecontrol, durham, mjpieters Reviewed By: mjpieters Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4571926 Signature: t1:4571926:1487263035:53419c9ca4e344127f391ac19c1071a826fea3f1
2017-02-16 21:18:22 +03:00
generation number: 0
fastpartialmatch: remove changectx.__init__() wrapper Summary: Turned out that wrapping `changectx.__init__()` doesn't give any performance benefits but instead it breaks initializing of changectx. For example, if there is a commit which revision number N is a prefix of another commit A then `hg log -r N` outputs commit A instead of commit N. Let's get rid of this wrapping. Test Plan: arc unit Reviewers: #mercurial, mjpieters Reviewed By: mjpieters Subscribers: mjpieters, #sourcecontrol Differential Revision: https://phabricator.intern.facebook.com/D4635311 Tasks: 16394593 Signature: t1:4635311:1488393427:eae091a0d155df336adbb122f2881a28e95ba6b6
2017-03-02 11:37:17 +03:00
file: 12, entries: 1, out of them 0 sorted
fastpartialmatch: trigger rebuilds if there were too many unsorted entries Summary: If we have too many unsorted entries then finding matching revisions is slower. To fix it we will rebuild partial index from time to time. _findcandidates function marks index as need-to-be-rebuilt if it notices index file with too many unsorted entries. Then _changegrouphook rebuilds the index. Test Plan: arc unit Reviewers: #sourcecontrol Differential Revision: https://phabricator.intern.facebook.com/D4494617
2017-02-02 16:12:10 +03:00
file: 3d, entries: 1, out of them 1 sorted
fastpartialmatch: fix the build Summary: Sort the output to fix tests on different platforms Test Plan: arc unit on mac Reviewers: #sourcecontrol, simonfar, ikostia Reviewed By: simonfar, ikostia Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4501449 Signature: t1:4501449:1486050577:3041dc3c3691f48e39a0e9e05b1f97954a6d88e2
2017-02-02 19:00:55 +03:00
file: 58, entries: 1, out of them 1 sorted
file: 64, entries: 1, out of them 1 sorted
fastpartialmatch: trigger rebuilds if there were too many unsorted entries Summary: If we have too many unsorted entries then finding matching revisions is slower. To fix it we will rebuild partial index from time to time. _findcandidates function marks index as need-to-be-rebuilt if it notices index file with too many unsorted entries. Then _changegrouphook rebuilds the index. Test Plan: arc unit Reviewers: #sourcecontrol Differential Revision: https://phabricator.intern.facebook.com/D4494617
2017-02-02 16:12:10 +03:00
file: ac, entries: 1, out of them 1 sorted
file: b7, entries: 1, out of them 1 sorted
file: fd, entries: 1, out of them 0 sorted
fastpartialmatch: fix devel.warn Summary: Upstream hg recently introduced new developer warning if files inside .hg are modified without taking a lock. The usual fix is to add file in localrepository._wlockfreeprefix list, but it doesn't work for "needrebuild" fastpartialmatch file because it's stored inside .hg/store. So to get rid of developer warning let's move needrebuild file to .hg/. And since we are detecting that partial index needs to be rebuilt in revlog class, I also had to wrap `runcommand()` and create needrebuild file there, because `runcommand()` has vfs object, while revlog class doesn't. Test Plan: Run tests Reviewers: #fbhgext, durham Reviewed By: #fbhgext, durham Subscribers: durham Differential Revision: https://phab.mercurial-scm.org/D99
2017-07-17 14:03:15 +03:00
Make a commit and change .hg permissions to non-writabble. Then do
fastpartialmatch: trigger rebuilds if there were too many unsorted entries Summary: If we have too many unsorted entries then finding matching revisions is slower. To fix it we will rebuild partial index from time to time. _findcandidates function marks index as need-to-be-rebuilt if it notices index file with too many unsorted entries. Then _changegrouphook rebuilds the index. Test Plan: arc unit Reviewers: #sourcecontrol Differential Revision: https://phabricator.intern.facebook.com/D4494617
2017-02-02 16:12:10 +03:00
partial lookup that should write needrebuild file but it couldn't because
of permissions. Make sure it doesn't throw and just log the problem
$ mkcommit commitpermissionissue
fastpartialmatch: fix devel.warn Summary: Upstream hg recently introduced new developer warning if files inside .hg are modified without taking a lock. The usual fix is to add file in localrepository._wlockfreeprefix list, but it doesn't work for "needrebuild" fastpartialmatch file because it's stored inside .hg/store. So to get rid of developer warning let's move needrebuild file to .hg/. And since we are detecting that partial index needs to be rebuilt in revlog class, I also had to wrap `runcommand()` and create needrebuild file there, because `runcommand()` has vfs object, while revlog class doesn't. Test Plan: Run tests Reviewers: #fbhgext, durham Reviewed By: #fbhgext, durham Subscribers: durham Differential Revision: https://phab.mercurial-scm.org/D99
2017-07-17 14:03:15 +03:00
$ chmod u-w .hg/
fastpartialmatch: trigger rebuilds if there were too many unsorted entries Summary: If we have too many unsorted entries then finding matching revisions is slower. To fix it we will rebuild partial index from time to time. _findcandidates function marks index as need-to-be-rebuilt if it notices index file with too many unsorted entries. Then _changegrouphook rebuilds the index. Test Plan: arc unit Reviewers: #sourcecontrol Differential Revision: https://phabricator.intern.facebook.com/D4494617
2017-02-02 16:12:10 +03:00
$ hg log -r . -T '{node}\n'
2b52832374dd7e499a1fbd172f1d75e13ee32477
$ hg log -r 2b5283237
changeset: 7:2b52832374dd
tag: tip
parent: 4:587cd78c6d0e
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: commitpermissionissue
fastpartialmatch: fix devel.warn Summary: Upstream hg recently introduced new developer warning if files inside .hg are modified without taking a lock. The usual fix is to add file in localrepository._wlockfreeprefix list, but it doesn't work for "needrebuild" fastpartialmatch file because it's stored inside .hg/store. So to get rid of developer warning let's move needrebuild file to .hg/. And since we are detecting that partial index needs to be rebuilt in revlog class, I also had to wrap `runcommand()` and create needrebuild file there, because `runcommand()` has vfs object, while revlog class doesn't. Test Plan: Run tests Reviewers: #fbhgext, durham Reviewed By: #fbhgext, durham Subscribers: durham Differential Revision: https://phab.mercurial-scm.org/D99
2017-07-17 14:03:15 +03:00
error happened while triggering rebuild: [Errno 13] Permission denied: '$TESTTMP/cloned2/.hg/partialindexneedrebuild'
$ chmod u+w .hg/
fastpartialmatch: add generation number Summary: fastpartialmatch index may be incorrect if fastpartialmatch extension was disabled, then new commits were added and then fastpartialmatch was enabled back. We need to detect this situation and rebuild the index in this case. One approach would be to use _handleinconsistentindex() function to mark index as invalid. There are a couple of drawbacks to this approach: 1) It requires more significant code changes 2) The output may still be incorrect if hash with the same prefix was added while fastpartialmatch was disabled. Another approach would be to use header version from index files (see class _header). That's even worse because we'll need to bump index version in the code whenever we disable fastpartialmatch. Third approach would be to use changelog mtime + size as a validation key but then we need to ensure partialmatch index is written after changelog. Instead I suggest to use generation numbers. It's just a number in the config and in the special file inside partialmatch index directory. If the two doesn't match then index is incorrect and needs to be deleted. The biggest drawback to this approach is that we need to bump generation number each time the extension is enabled. I think it should be fine because the one who enables extensions usually understands it and he or she will bump the generation number. Test Plan: arc unit Reviewers: #sourcecontrol, durham, mjpieters Reviewed By: mjpieters Subscribers: mjpieters Differential Revision: https://phabricator.intern.facebook.com/D4571926 Signature: t1:4571926:1487263035:53419c9ca4e344127f391ac19c1071a826fea3f1
2017-02-16 21:18:22 +03:00
Temporarily disable the fastpartialmatch and make a commit.
Bump index generation number version and check that index was deleted after next
invocation
$ echo disabled > disabled
$ hg add disabled
$ hg --config extensions.fastpartialmatch=! ci -m disabled
$ hg debugcheckpartialindex
e26f27bfb47a8eb9df0ee8f0feeb850722bc416d node not found in partialindex
[1]
$ printf "\n[fastpartialmatch]\ngenerationnumber=1\n" >> .hg/hgrc
$ hg log -r . -T '{node}'
e26f27bfb47a8eb9df0ee8f0feeb850722bc416d (no-eol)
$ hg debugcheckpartialindex
partial index is not built
[1]
Make pull to rebuild the index
$ cd ../repo
$ mkcommit servercommit
$ cd ../cloned2
$ hg pull -q
$ hg debugcheckpartialindex
$ hg debugfastpartialmatchstat
generation number: 1
file: 06, entries: 1, out of them 0 sorted
file: 12, entries: 1, out of them 1 sorted
file: 2b, entries: 1, out of them 1 sorted
file: 3d, entries: 1, out of them 1 sorted
file: 58, entries: 1, out of them 1 sorted
file: 64, entries: 1, out of them 1 sorted
file: ac, entries: 1, out of them 1 sorted
file: b7, entries: 1, out of them 1 sorted
file: e2, entries: 1, out of them 1 sorted
file: fd, entries: 1, out of them 1 sorted
Write incorrect generation number
$ echo badgennum > .hg/store/partialindex/generationnum
$ hg log -r .
error happened while reading generation num: invalid literal for int() with base 10: 'badgennum\n'
changeset: 8:e26f27bfb47a
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: disabled
$ hg debugcheckpartialindex
partial index is not built
[1]
fastpartialmatch: remove changectx.__init__() wrapper Summary: Turned out that wrapping `changectx.__init__()` doesn't give any performance benefits but instead it breaks initializing of changectx. For example, if there is a commit which revision number N is a prefix of another commit A then `hg log -r N` outputs commit A instead of commit N. Let's get rid of this wrapping. Test Plan: arc unit Reviewers: #mercurial, mjpieters Reviewed By: mjpieters Subscribers: mjpieters, #sourcecontrol Differential Revision: https://phabricator.intern.facebook.com/D4635311 Tasks: 16394593 Signature: t1:4635311:1488393427:eae091a0d155df336adbb122f2881a28e95ba6b6
2017-03-02 11:37:17 +03:00
Create bookmark with the same prefix as commit hash. hg log should show commit
with bookmark
$ hg debugrebuildpartialindex
$ hg book e26f27bf
fix tests after upstream change b2eb5bb9709893ebd015b97c06f4242b3b4c7b48 in upstream broke our tests. This commit fixes it
2017-05-25 19:55:51 +03:00
bookmark e26f27bf matches a changeset hash
(did you leave a -r out of an 'hg bookmark' command?)
fastpartialmatch: remove changectx.__init__() wrapper Summary: Turned out that wrapping `changectx.__init__()` doesn't give any performance benefits but instead it breaks initializing of changectx. For example, if there is a commit which revision number N is a prefix of another commit A then `hg log -r N` outputs commit A instead of commit N. Let's get rid of this wrapping. Test Plan: arc unit Reviewers: #mercurial, mjpieters Reviewed By: mjpieters Subscribers: mjpieters, #sourcecontrol Differential Revision: https://phabricator.intern.facebook.com/D4635311 Tasks: 16394593 Signature: t1:4635311:1488393427:eae091a0d155df336adbb122f2881a28e95ba6b6
2017-03-02 11:37:17 +03:00
$ mkcommit newcommitwithbook
$ hg log -r e26f27bf
changeset: 10:bf72b6cd3f5a
bookmark: e26f27bf
tag: tip
parent: 8:e26f27bfb47a
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: newcommitwithbook
fastpartialmatch: add amended commits to the index Summary: Upstream mercurial doesn't run hooks for temporary amended commits (see https://bz.mercurial-scm.org/show_bug.cgi?id=3501 and c2ca20984e3c9f30d73f0f35c35904b64edbd692). No other hook contains info about temporary amended commits. Let's wrap localrepository.commit() function and add missing commits inside this wrapper. Test Plan: arc unit Reviewers: #mercurial, durham Reviewed By: durham Subscribers: durham, mjpieters, #sourcecontrol Differential Revision: https://phabricator.intern.facebook.com/D4634950 Tasks: 16387175 Signature: t1:4634950:1488399457:1d299c6ad3cfd418dda32a7ec20c60290f8d7778
2017-03-02 12:14:09 +03:00
Do amend and check index
$ hg debugrebuildpartialindex
$ mkcommit toamend
$ echo amended > toamend
$ hg ci -m amended --amend -q
$ hg debugcheckpartialindex
Try to create empty commit
$ hg ci -m empty
nothing changed
[1]
fastpartialmatch: handle nullid correctly Summary: Special case for nullid Test Plan: arc unit Reviewers: #mercurial, simpkins, ikostia Reviewed By: ikostia Subscribers: net-systems-diffs@fb.com, mjpieters, #sourcecontrol Differential Revision: https://phabricator.intern.facebook.com/D4945927 Tasks: 17622253 Signature: t1:4945927:1493123209:712990c1e883241d99c0aeb4af8d48e18e1baa3d
2017-04-26 13:55:59 +03:00
Try to checkout nullid
$ hg up --config fastpartialmatch.raiseifinconsistent=True -q 0000000000
Reference in New Issue Copy Permalink