Upstream added __bool__ to manifestdict, and the test checking that
hybridmanifest matches the functions of manifestdict started failing. Let's add
it.
The check-code.py change matches the one in upstream to support the addition of
__bool__.
Summary:
Sometimes the cache directory has wrong group set and our hg code fails with
permission errors. Try to solve that by detecting wrong groups or modes and
reset them.
Test Plan:
```
In [2]: from remotefilelog import shallowutil
In [5]: ui.setconfig('remotefilelog', 'cachegroup', 'kvm')
In [6]: shallowutil.mkstickygroupdir(ui, '/tmp/d1/d2')
# make sure /tmp/d1, /tmp/d1/d2 have group=kvm and the sticky bit set.
# run `sudo chown -R quark:quark /tmp/d1`
# run `sudo chmod g-s -R /tmp/d1`
In [7]: shallowutil.mkstickygroupdir(ui, '/tmp/d1/d2')
# make sure /tmp/d1/d2 is owend by "kvm" group and has the sticky bit
# again, while /tmp/d1 remains unchanged.
```
Reviewers: #sourcecontrol, stash
Reviewed By: stash
Subscribers: stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4701627
Tasks: 16473317
Signature: t1:4701627:1489481138:64a6038ec5cc067cf05bad3b14ee9985e0bf6d96
Summary:
It will allow sparse to not ignore the file, and is more consistent with
other special files.
Test Plan: Updated existing tests
Reviewers: rmcelroy, #sourcecontrol, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4738766
Signature: t1:4738766:1490032493:199f3fef9c74a137b16ae7637b87de625ca5115d
Summary:
ui.shortuser() doesn't call util.shortuser() if ui.verbose equals to True.
But in our case it's very important to always use short username because
we'll get broken backups otherwise. Let's call util.shortuser() directly.
Test Plan: arc unit
Reviewers: #mercurial, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4721949
Tasks: 15389402
Signature: t1:4721949:1489717973:b7e59929f530578e060b8cdef94ef92b54fa2647
Summary:
We forgot one more store file: obsinhibit.
This diff adds it
Test Plan: arc unit
Reviewers: #mercurial, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4714522
Tasks: 16558840
Signature: t1:4714522:1489717585:fdabf3c5f0c11310708ba3d78df39cbaa04a3afa
Summary:
This diff just restores the code that was reverted before. Next diffs will fix
it
Test Plan: arc unit
Reviewers: #mercurial, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4714397
Tasks: 16558840
Signature: t1:4714397:1489717436:9777d783d1443dea46460450e8f3da862c147483
Summary:
There was a bug where if a commit had an empty manifest (i.e. same contents as
it's parents, but different p1/p2) and it resulted in a
non-empty-but-still-a-noop delta in the revlog (i.e. a delta that deletes a line
and replaces it with the same content), this resulted in a no-op set to the tree
manifest. When the tree was serialized, it noticed that the set was a no-op, so
it didn't serialize that particular tree, but the parent didn't get notified it
was a no-op, so we serialized parent directories with pointers to sub trees that
did not exist.
The fix is to not store new sub-tree nodes on parents when the sub-tree contents
are the same. Now we just store the original sub-tree node. So we no longer
accidentally reference non-existent trees.
Unfortunately I'm not sure how Mercurial can get into this situation (how do you
produce a delta that has content, but the content is a no-op?), so I'm not sure
how to test it. The tree verification command in another patch can catch this
exception though.
Test Plan:
Ran 'hg debuggentrees' on a repo that has a manifest entry that
exhibits this problem. Verified via the debugger that only one tree (the root
node) was generated from adding that manifest.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4724539
Signature: t1:4724539:1489789531:02a9a75a85aa2a0a6e4c16e163867bd5a6f55670
Summary:
This code that reused deltas if the delta parent wasn't available was bugged
because it meant you could end up with a cycle in the delta chains. This was an
old optimization from before trees had history, so let's drop the optimization
(since trees now have history and can be correctly repacked).
Test Plan:
Ran repack on a packfile that previously caused cycles. Verified the
new version did not with `hg debugdatapack foo.datapack'
Reviewers: #mercurial
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4724520
Summary:
This adds 'hg debuggentrees' that takes an optional --verify flag. This makes it
easier to debug tree creation and catch errors.
Test Plan:
Ran `PYTHONPATH=/data/users/durham/fb-hgext python ../hg/hg --pager=off debuggentrees --verify master~xxxx master` to generate 6 months
worth of trees. It found bugs which are fixed in other patches.
Reviewers: #mercurial
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4724502
Summary:
It'd be nice to have dirsync config in the repo, since it's better synced, and
easier to be modified.
Test Plan: Added a test case
Reviewers: #sourcecontrol, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4732054
Signature: t1:4732054:1489793650:ca6c63ef47a1d9ebd181fbaee5554975801324eb
Summary:
Upstream converted all array.array usages to bytearray, and in doing so they
also changed mdiff.textdiff(array1, array2) to be
mdiff.textdiff(util.buffer(array1), util.buffer(array2)), which we did not do in
our internal implementation.
This fixes that.
Test Plan: Adds a test
Reviewers: #mercurial, quark, simonfar
Reviewed By: quark, simonfar
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4724350
Signature: t1:4724350:1489697253:8f63350bfc30b3079bf38410054b4763942a82f6
Summary:
On macos sshpeer flushes remote output later and it causes tests to fail. Let's
add a call to cleanup() to make sure the output is flushed.
This diff also makes remotefilelog test more robust.
Test Plan: Run tests on macos
Reviewers: #mercurial, tja, mitrandir
Reviewed By: mitrandir
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4721276
Tasks: 15389402
Signature: t1:4721276:1489669109:4ce59f4a1d224d57dbb7c1eb341c4e6a659d2e8c
Summary:
By default debugcheckbackup checks only one backup for the user.
With --all option it checks all backups for the user from all host/reporoots
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4700993
Tasks: 15389402
Signature: t1:4700993:1489548921:800d08f420acc8ef4f807ffd17b31837dbb3fe82
Summary:
Backing up very big repo can cause problems. And very often old commits are
not necessary at all. Let's add a config option to limit the number of heads
that are being backed up.
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4700965
Tasks: 15389402
Signature: t1:4700965:1489549163:4e2c121f01debd7b495486a1f3b062926873399d
Summary:
Previously we saved bookmark hash and revision number as state of
the last backup. Storing last backed up revision is error-prone. I saw
a few corrupted backups where bookmarks pointed to non-backed up
nodes. Also this approach puts more pressure on mysql bundle index.
On every backup it first deletes all existing backup bookmarks and then it
writes new bookmarks. Even if user changed one head or one bookmark
(and that is the usual case) current approach still deletes bookmarks first
and then rewrites them.
Instead let's store all backed up heads and bookmarks locally in json format.
Json was chosen because it's simple and human-readable. Then during backup
let's compare last backed up state to the current state of the repo, and send
to the server only the difference.
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4700930
Tasks: 15389402
Signature: t1:4700930:1489613249:a34369bf53e0718c22258304493dfa27b578857f
Summary:
Let's rename _getbackupstate() to _downloadbackupstate() because it actually
downloads it from the server. Also return class instead of tuple
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4713633
Tasks: 15389402
Signature: t1:4713633:1489612870:3439a6a99ef311bf96784b3b1e53d7e9b94c0713
Summary:
This adds ui.log() output for prefetch statistics. Extensions who hook into
ui.log() can now log this data to external metrics systems.
Test Plan:
Ran a hg prefetch with the config flags enabled, while ptailing the dev command
timer. Verified the result contained remotefilelogfetches*
```
CHGDISABLE=1 FB_HG_DIAGS=1 hg --config
extensions.remotefilelog=../fb-hgext/remotefilelog/ --config
sampling.key.remotefilelog.prefetch=perfpipe_dev_command_timers prefetch -r .~9
ptail -f perfpipe_dev_command_timers | grep durham
```
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4711096
Signature: t1:4711096:1489591144:1c91a4fbd118a3c10c2a2c68391c9f5b0dbcedf3
Summary:
We have encountered a kernel issue where `rmdir` a non-empty directory may race
with other things and hang in kernel for a long time.
This patch changes `os.rmdir` to avoid `rmdir` non-empty directories. It is
written in Cython calling the low-level `readdir` libc friends to make overhead
minimal.
Test Plan: Added a new test
Reviewers: #sourcecontrol, clm, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, simpkins, osandov, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4716711
Tasks: 16647532
Signature: t1:4716711:1489627923:7c7432748c1fd8c070ce257bd172feebd3807f65
test-check-code doesn't like sed -i, so let's switch to a tiny batch script.
Also fix the test importing the extension directly instead of relying on the
path to be right.
Summary:
It's just copy-paste from my mutable-history patch series that ended up as
persistent hotfix.
The wrapping logic is very simple: it redirects '--fold' calls to original
metaedit and handles all the rest.
Test Plan:
Included a big copy-pase from test-evolve.t preserving only cases that were
executing metaedit and modified for the smaller test repo (the metaedit tests
were at the end of evolve test file).
Reviewers: #sourcecontrol, quark, simonfar
Reviewed By: quark, simonfar
Subscribers: rmcelroy, quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4708017
Signature: t1:4708017:1489524494:d682aab857e8422bb1a23adc22dddfb9f505b6b3
Summary: New logging will have rotation and will separate different users and repos
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4689191
Tasks: 16474976
Signature: t1:4689191:1489172091:d304d4b19ae6cf52f86c11f442fa760b50d1fdf9
Summary:
The upstream has changed how color works. `chistedit` seems to use a very hacky
way to disable color. Let's use a safer and simpler approach.
Test Plan: Run chistedit with hg-committed.
Reviewers: #sourcecontrol, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4702229
Signature: t1:4702229:1489446376:f0c7b4d31eb62bb6aea4cfeb13d78be8c219374c
Summary:
Upstream has switched the type here for better python 3 compatibility. Let's
match their change.
Test Plan: Tests now pass
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4699371
Signature: t1:4699371:1489429276:9da9f64489ecaf94cd77599a0558c081cfe792c2
Summary: The poorly formatted docstring was causing an extra blank line to be inserted both before and after the content
Test Plan: Changed a binary in an hg repo. Committed interactively using the editor. Made sure that there were no blank lines
Reviewers: rmcelroy, quark, simonfar
Reviewed By: simonfar
Subscribers: most, asriram, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4684549
Tasks: 15728734
Signature: t1:4684549:1489269897:faecd381c98e8c2b9884f9bacd18dfc32ee77758
Summary:
For now these command will be used in test because
`wait_for_background_backup.py` tied to the backup state file. And every change
to this file requires change to wait_for_background_backup.py. Let's create
simple internal backup command.
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4667056
Tasks: 15389402
Signature: t1:4667056:1489024344:6c43241e42b7c418baa4e1542f303239ba887c45
Summary:
There can be multiple backup processes running at the same time. Since these
processes can be quite heavy it makes sense to limit them. Let's use lock file
to do that.
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4667020
Tasks: 15389402
Signature: t1:4667020:1489170813:f2a685b1c224c553d3ee004d89d3eeeca816e824
Summary:
Previously, the rebase logic itself would unlink its state file. Now that the
state file is part of the transaction, it gets serialized at the end of the
transaction, even if it was unlinked midway through the transaction. This
affects fbamend because it wraps various rebases in a higher transaction.
Ideally we would add support to the mercurial transaction framework to allow
marking a file as deleted halfway through the transaction, but this patch will
fix the tests until we get that upstream.
Test Plan: Tests now pass
Reviewers: kulshrax, #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4696578
Signature: t1:4696578:1489348040:b12104b2519d69dab98d2076f640f9f3157036e2
Summary: Upstream has added a new attribute to dirstate. Let's add it too.
Test Plan: Tests now pass
Reviewers: mitrandir, #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4695923
Signature: t1:4695923:1489283439:c7876832af2764d2bbe9cd0f9e47199c84799609
Summary:
Upstream changed the format from ['diff ...\n', 'file1\nfile2\nchunks...'] to be
['diff ...\nfile1\nfile2\n', 'chunks...'] in 199440c7d1f49752. So grepdiff needs
to be updated.
Test Plan: Tests now pass
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4695911
Signature: t1:4695911:1489282635:920e88fecc5d6d4660d1d21289d1d07e26c93315
Summary:
Upstream has added a matcher arg to manifest.filesnotin, so let's support it in
our native treemanifest implementation.
Test Plan: Added unit tests for filesnotin with and without matchers.
Reviewers: #mercurial, stash
Reviewed By: stash
Subscribers: stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4677030
Signature: t1:4677030:1489076599:23ab67d85bb4fa689f05349253a4d2c4e7d496cc
Summary:
Upstream has added a matcher argument to the diff API which allows diff to avoid
traversing certain parts of the tree. This adds support for that to our native
treemanifest implementation.
Test Plan: Added tests for diff with matches
Reviewers: #mercurial, stash
Reviewed By: stash
Subscribers: stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4677023
Signature: t1:4677023:1489076627:dbcea209d300a68fa050f68c52b4fd9949b85302
Summary:
A bunch of if statements were doing the same thing every time. This moves that
logic out of the individual if statements, which makes it cleaner and will make
a subsequent patch that runs a matcher against the paths easier.
Test Plan: Ran the tests
Reviewers: #mercurial, stash
Reviewed By: stash
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4677011
Signature: t1:4677011:1489075541:9379597b8866358ad5dad3f1f9ae00a0a0b523f9
Summary: Let's add option that contains a list of other config options to log.
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4681336
Signature: t1:4681336:1489171115:dddd27032e945f37fcc846a511e3aaa9e83d29d3
Summary:
Upstream has added a new match argument to manifest.diff() and removed the
existing manifest.matches() function, so we need to update our internal usage.
A separate diff will update treemanifest to support the new diff() api.
Test Plan:
Ran the tests, some still fail because of the upstream changes, but
future patches fix those.
Reviewers: #mercurial
Differential Revision: https://phabricator.intern.facebook.com/D4677002
Summary: This diff adds a `--towards` flag to `hg next`. This flag tells `hg next` to always proceed in a linear fashion towards the specified commit. This is useful in situations where the user has a very branchy stack. For example, if the user has a local stack with several branches, each with a feature bookmark at the top, if the user wants to go up a particular stack they can do so with something like `hg next 3 --towards bookmark`.
Test Plan:
Here is an example of basic usage. Also see test file.
```
$ hg debugbuilddag "+5 *2 +2"
$ hg book -r 4 feature1
$ hg book -r 7 feature2
$ hg up 0
$ hg sl
o 92eaf3 debugbuilddag feature2
| r7
|
o 8bbe84 debugbuilddag
| r6
|
o 914970 debugbuilddag
| r5
|
| o bebd16 debugbuilddag feature1
| | r4
| |
| o 2dc09a debugbuilddag
| | r3
| |
| o 012414 debugbuilddag
|/ r2
|
o 66f7d4 debugbuilddag
| r1
|
@ 1ea734 debugbuilddag
r0
$ hg next 3 --towards feature1
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
[2dc09a] r3
$ hg sl
o 92eaf3 debugbuilddag feature2
| r7
|
o 8bbe84 debugbuilddag
| r6
|
o 914970 debugbuilddag
| r5
|
| o bebd16 debugbuilddag feature1
| | r4
| |
| @ 2dc09a debugbuilddag
| | r3
| |
| o 012414 debugbuilddag
|/ r2
|
o 66f7d4 debugbuilddag
| r1
|
o 1ea734 debugbuilddag
r0
```
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4695861
Signature: t1:4695861:1489325959:1145a29ba87f99d07ede1e81415804723dc48818
Summary:
We are relying on pushrebase to replay each of the incoming commits so they
produce both flat and treemanifests. This patch blocks any commits that don't
come through pushrebase, by intercepting manifest.addgroup calls (which is what
a normal push would use) and throwing an exception.
Test Plan: Added a test
Reviewers: #mercurial, stash
Reviewed By: stash
Subscribers: stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4093745
Signature: t1:4093745:1478100170:79b1eea6668cd3efb27742323e11a45bde930440
Summary:
When a server is a treemanifest enabled server, we want to ensure that every
commit that comes in will be added to the treemanifest as well as the flat
manifest. This patch accomplishes that by hooking into the addmanifest logic and
applying the new manifest to the tree as well.
Test Plan: Added a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4093738
Signature: t1:4093738:1478260442:58f77bd88ab09ccda4ac238a065388be8f4a271d
Summary:
This adds a command for producing tree manifests for existing manifests in the
repository. The tree manifests are stored similar to how vanilla mercurial
stores them, but the root revs are in 00manifestree.i instead of 00manifest.i
and we fake the root node to be the same as the flat manifest node, so we can
reference either the flat or the tree easily.
Test Plan: Added a test. Also began converting fbsource
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4092252
Signature: t1:4092252:1478260394:a55360a8ba83ad85b1694cb93f0788f1f672ef61
Summary:
As part of the migration, we need both flat and tree manifests to exist at once
on the server. To do this, we need to store the tree manifest root revisions
somewhere other than 00manifest.i.
This patch divides the treemanifest logic into client and server bits, and on
the server creates a repo.manifesttree property that contains the tree manifest.
Future patches will write to this to fill in new tree entries, and eventually
read from it to seed clients with tree data.
Test Plan:
Future patches add tests that for features that use this
functionality
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4092217
Signature: t1:4092217:1478260318:7edada2ea39a5de565955a4180bfad62f6126352
