Summary:
Real infinitepush tests are in fb-hgext/tests. These tests are useless and we
never run them. Let's remove them.
Test Plan: arc unit
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4778858
Signature: t1:4778858:1490700082:69bcd2adee2a93ec5d3886747d47bc30b18f8944
Summary:
Previously we don't set findcommonincoming to True, and that bug was unnoticed
because we send this info anyway in server-side `getbundlechunks()` function.
But next diff in the stack uses a fast path server-side which won't work if
fincommonincoming is not set to True.
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4770830
Tasks: 15389402
Signature: t1:4770830:1490719209:c251002d992e244580b7dbfeca0c30dff95734d2
Summary:
Command to list available backups for the user.
--json will be used by automation
Test Plan: arc unit
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4728729
Tasks: 15389402
Signature: t1:4728729:1490005762:41b9683cb7dfc9d84ae1032c807d1f0c3fe60dbf
Summary:
Only match the "Differential Revision" label at the start of a line. We
have some diffs that include legitimate-looking Differential Revision label
strings inside parts of their test plan, which previously confused the phabdiff
output.
Test Plan: Included a unit test.
Reviewers: #sourcecontrol, quark, akushner
Reviewed By: akushner
Subscribers: net-systems-diffs@fb.com, yogeshwer, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4793483
Signature: t1:4793483:1490756514:05b2c49d5d65e1a3eadd5eb78ed0b91ef3bea42c
Summary:
From diff:
```
This test does two things:
1/ Simulate a common condition of pushrebase under load. Normally pushrebase
caches data before acquiring the write lock (if lazy locking is enabled).
Under load, however, when a push has to wait for the lock more often than not,
much of this data becomes out of date and must be refetched once the lock is
acquired. This test simulates that particular case.
Specifically, we create two clients, client1 and client2, both with
nonconflicting changesets to push. client1's push is artificially blocked by a
`prepushrebase` hook (post-caching, pre-lock) that is only released after
client2's push succeeds.
2/ Checks how often we call manifest.read() inside the lock (and outside).
This way we can prevent regressions on manifest reads and test improvements.
manifest.read() is wrapped by an extension that prints a short trace. read calls
inside the lock are marked with a ":(".
```
This test is a bit of a doozy, simplification suggestions are encouraged.
Test Plan: this is, in fact, a test
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: quark, rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4785464
Signature: t1:4785464:1490737637:79d77f6b2e49a428d63b8e1b13d769c82149ee72
Summary:
Previously we copied the old bundle dirmancache entries into the new bundle
dirmancache entries, but that missed any manifests that weren't already in the
cache. Since we have a list of bundle revs that are guaranteed to be needed
during the rebase, lets read them before the lock and force them into the cache
after the lock.
Test Plan:
Ran all pushrebase tests with a block like:
```
bundle.manifestlog[mfnode].read()
realmf = newdirmancache[''][mfnode].read()
if realmf.text() != mfdict.text():
raise Exception("cache invalid")
```
...to verify the manifest we cached is the same as can be read from disk.
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4774728
Signature: t1:4774728:1490736995:6e20d69181b4df76aeb800ea246a0dae0a4b802d
Summary:
This fetches the old master's full text before the lock is taken, and puts it in
the revlog._cache field so we don't have to read the full delta chain when we go
to read the latest master revision. This minimizes the amount of time we have to
spend processing the manifest during the first write.
Test Plan:
Ran all tests using a block like the following:
```
for mfnode, mfdict in bundlerepocache.iteritems():
bundle.manifestlog[mfnode].read()
existing = newdirmancache[""][mfnode].read().text()
new = mfdict.text()
if new != existing:
raise Exception("Cache failed")
```
...which raises if the data we were about to pull from the cache was different from what we were reading from the manifest before.
Reviewers: #mercurial, durham, rmcelroy
Reviewed By: durham
Subscribers: rmcelroy, quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4774716
Signature: t1:4774716:1490735677:85bac79999d607f5554d090978d0ee6787f5e11c
Summary:
Previously, pushrebase detected if it was ok to rebase a commit by diff'ing the
original common ancestor of the bundle and master with the new master. This
required reading both manifests. To improve performance, let's read the
changelog files list instead, which is much, much faster.
Merge commits don't have a files list that is representative of the actual diff,
so fall back to manifest diff in that case.
Test Plan: ran tests
Reviewers: #mercurial, stash, durham
Reviewed By: stash, durham
Subscribers: stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4774694
Signature: t1:4774694:1490601056:18f78ab7c3e35a1beb2a7775160271a3311e38bb
Summary:
Previously `lfs.blobstore` must be a relative path. If an absolute
blobstore path is set, it will traceback because vfs audit fails:
```
File "hg/mercurial/revlog.py", line 1356, in _processflags
text, vhash = writetransform(self, text)
File "fb-hgext/hgext3rd/lfs/wrapper.py", line 69, in writetostore
blobstore.local.get(self.opener).write(storeid, chunk)
File "fb-hgext/hgext3rd/lfs/blobstore.py", line 41, in write
fp = self._opener(self.filename(storeid), 'w+', atomictemp=True)
File "hg/mercurial/vfs.py", line 344, in __call__
self.audit(path)
File "hg/mercurial/pathutil.py", line 64, in __call__
raise error.Abort(_("path contains illegal component: %s") % path)
Abort: path contains illegal component: /home/quark/lfslocalblobstore/d7/dbc611df1fe7dfacfe267a2bfd32ba8fc27ad16aa72af7e6c553a120b92f18
```
That was because the code was using `repo.vfs`. This diff adds a new `lfsvfs`
to avoid the issue. The `lfsvfs` also did the correct filename check (the
old `re.match` check will not match the whole string), so `blobstore.local`
could be simplified a lot.
Test Plan:
A new test case was added to make sure absolute blobstore path works. I also
did some cleanups for the test file to de-dup hgrc, and avoid writing files
outside `$TESTTMP`.
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters, remi
Differential Revision: https://phabricator.intern.facebook.com/D4785084
Signature: t1:4785084:1490693306:b42eef8e53af107897c2e1bc2984c090bdd2e465
Summary:
The core mercurial could now detect filesystem for Linux and OSX, and enable
real hardlink support for their major filesystems. So the hardlink tweak is no
longer needed.
Test Plan: `arc unit`
Reviewers: #mercurial, stash
Reviewed By: stash
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4776139
Signature: t1:4776139:1490601097:0be66e2a0111b71a932b453ef4e1e75fed8276e6
Summary:
It's the upstream mutable-history that handles the "--fold" case now.
It doesn't have the fix that avoids touching wd in that case.
Test Plan:
tests are passing when mutable-history is updated to default-upstream-tip
I'll delete the actual mutable-history hotfixes once this is accepted.
Reviewers: #sourcecontrol
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4779366
Summary:
A bunch of modifications to get the test pass with the new lfs code.
- Move `lfs` to `hgext3rd`. The code was supposed for hg-core. For now, we do
them in fb-hgext to speed up the process
- Remove the windows test, which is not supported by `run-tests.py` and is
duplicated with `test-lfs.t`.
- Do import `mercurial.i18n._` correctly.
- Change some i18n logic a bit so it's more translator-friendly.
- Change `revlog.RevlogError` to `error.RevlogError`.
- Avoid direct symbol import of `mercurial.util.bytecount`, which will fail the
upstream importchecker test.
- Fix various lint issues like lines being too long etc.
- Document lfs config options.
Test Plan: `arc unit`
Reviewers: #sourcecontrol, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4772216
Signature: t1:4772216:1490401458:1ad3c18ab80e1d31085d0b6b4c630e62a7dc7930
Summary: test-check-code.t now dislikes our docstrings. Fix them up
Test Plan: Run the test locally on my devserver
Reviewers: #sourcecontrol, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4773908
Signature: t1:4773908:1490394131:9d9e5f85b4243119a3615044605bcc5abdecbe4f
Summary: `resource` is UNIX-only module, we can't use it if we want `pushrebase` to work on Windows. As we only use `getpagesize()`, we can also do `mmap.PAGESIZE` instead.
Test Plan: - apply change on Linux, see that they do not introduce any new failures
Reviewers: #sourcecontrol, durham, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4762553
Signature: t1:4762553:1490294453:fdcb2e9caa38f5d14eafc6ff36a1bb7d9429b6e9
Summary: Just as in D4762189. Fixes to match upstream changes
Test Plan: arc unit
Reviewers: #sourcecontrol, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4762596
Signature: t1:4762596:1490286227:004ed0a7055aab64769893e80a995975b62d5ea0
Summary:
changeset: 06d6fecfdb713e3a2e2013de046af0bce4005b96 (@)
user: Jun Wu <quark@fb.com>
date: Mon, 13 Mar 2017 22:19:06 -0700
histedit: use safecleanupnode in _finishhistedit
This simplifies code a lot.
causes significant changes to the test output. Update to match
Test Plan: Run tests locally
Reviewers: #sourcecontrol, quark, stash
Reviewed By: stash
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4762189
Signature: t1:4762189:1490281032:6a48429f60584c4318bd1e247ce2eeb2775d623c
Summary:
This follows the removal of urlparse upstream:
changeset: 97aa316e17ea7a0d3ba4180b23ef29bd32e409fc
user: Gregory Szorc <gregory.szorc@gmail.com>
date: Tue, 21 Mar 2017 22:47:49 -0700
py3: stop exporting urlparse from pycompat and util (API)
Test Plan:
./run-tests.py locally
Copy extension into place on my devserver and confirm that phrevset still works in www:
04:27:55 simonfar@devvm022:~/www (bb5bd09) $ hg show D1234
D1234 does not have an associated version control system
You can view the diff at http://phabricator.fb.com/D1234
abort: unknown revision D1234
04:28:53 simonfar@devvm022:~/www (bb5bd09) $ hg show D4721979
changeset: 74f087400ed442941d7400749d7f0301f7b6662a D4721979
user: alonsch@2c7ba8d8-a2f7-0310-a573-de162e16dcc7
date: Mon, 20 Mar 2017 04:07:37 -0700
[Portal] Remove outdated TODO comments in OnavoLineChart
Reviewers: #sourcecontrol, ikostia
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4761905
Summary:
I accidently disabled fastmanifest and a bunch of other packages on non-win
platforms, instead of Windows itself. This should fix it.
Test Plan: - makes sense visually
Reviewers: #sourcecontrol, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4761873
Signature: t1:4761873:1490267573:74e81c97652f15b06b7b7a56f96bf3d35385f54e
As far as I can tell, 'nodechunk' is internal to remotefilelog (i.e. this
should not be called by mercurial directly) and every callsite has nodes
instead of revisions here.
Summary:
Hardlink backups are significantly cheaper than file copies. They are
disabled in core because they are problematic on CIFS.
Test Plan: Ran the tests.
Reviewers: #sourcecontrol
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4652490
Tasks: 16406729
Upstream added __bool__ to manifestdict, and the test checking that
hybridmanifest matches the functions of manifestdict started failing. Let's add
it.
The check-code.py change matches the one in upstream to support the addition of
__bool__.
Summary:
Sometimes the cache directory has wrong group set and our hg code fails with
permission errors. Try to solve that by detecting wrong groups or modes and
reset them.
Test Plan:
```
In [2]: from remotefilelog import shallowutil
In [5]: ui.setconfig('remotefilelog', 'cachegroup', 'kvm')
In [6]: shallowutil.mkstickygroupdir(ui, '/tmp/d1/d2')
# make sure /tmp/d1, /tmp/d1/d2 have group=kvm and the sticky bit set.
# run `sudo chown -R quark:quark /tmp/d1`
# run `sudo chmod g-s -R /tmp/d1`
In [7]: shallowutil.mkstickygroupdir(ui, '/tmp/d1/d2')
# make sure /tmp/d1/d2 is owend by "kvm" group and has the sticky bit
# again, while /tmp/d1 remains unchanged.
```
Reviewers: #sourcecontrol, stash
Reviewed By: stash
Subscribers: stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4701627
Tasks: 16473317
Signature: t1:4701627:1489481138:64a6038ec5cc067cf05bad3b14ee9985e0bf6d96
Summary:
It will allow sparse to not ignore the file, and is more consistent with
other special files.
Test Plan: Updated existing tests
Reviewers: rmcelroy, #sourcecontrol, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4738766
Signature: t1:4738766:1490032493:199f3fef9c74a137b16ae7637b87de625ca5115d
Summary:
ui.shortuser() doesn't call util.shortuser() if ui.verbose equals to True.
But in our case it's very important to always use short username because
we'll get broken backups otherwise. Let's call util.shortuser() directly.
Test Plan: arc unit
Reviewers: #mercurial, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4721949
Tasks: 15389402
Signature: t1:4721949:1489717973:b7e59929f530578e060b8cdef94ef92b54fa2647
Summary:
We forgot one more store file: obsinhibit.
This diff adds it
Test Plan: arc unit
Reviewers: #mercurial, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4714522
Tasks: 16558840
Signature: t1:4714522:1489717585:fdabf3c5f0c11310708ba3d78df39cbaa04a3afa
Summary:
This diff just restores the code that was reverted before. Next diffs will fix
it
Test Plan: arc unit
Reviewers: #mercurial, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4714397
Tasks: 16558840
Signature: t1:4714397:1489717436:9777d783d1443dea46460450e8f3da862c147483
Summary:
There was a bug where if a commit had an empty manifest (i.e. same contents as
it's parents, but different p1/p2) and it resulted in a
non-empty-but-still-a-noop delta in the revlog (i.e. a delta that deletes a line
and replaces it with the same content), this resulted in a no-op set to the tree
manifest. When the tree was serialized, it noticed that the set was a no-op, so
it didn't serialize that particular tree, but the parent didn't get notified it
was a no-op, so we serialized parent directories with pointers to sub trees that
did not exist.
The fix is to not store new sub-tree nodes on parents when the sub-tree contents
are the same. Now we just store the original sub-tree node. So we no longer
accidentally reference non-existent trees.
Unfortunately I'm not sure how Mercurial can get into this situation (how do you
produce a delta that has content, but the content is a no-op?), so I'm not sure
how to test it. The tree verification command in another patch can catch this
exception though.
Test Plan:
Ran 'hg debuggentrees' on a repo that has a manifest entry that
exhibits this problem. Verified via the debugger that only one tree (the root
node) was generated from adding that manifest.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4724539
Signature: t1:4724539:1489789531:02a9a75a85aa2a0a6e4c16e163867bd5a6f55670
Summary:
This code that reused deltas if the delta parent wasn't available was bugged
because it meant you could end up with a cycle in the delta chains. This was an
old optimization from before trees had history, so let's drop the optimization
(since trees now have history and can be correctly repacked).
Test Plan:
Ran repack on a packfile that previously caused cycles. Verified the
new version did not with `hg debugdatapack foo.datapack'
Reviewers: #mercurial
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4724520
Summary:
This adds 'hg debuggentrees' that takes an optional --verify flag. This makes it
easier to debug tree creation and catch errors.
Test Plan:
Ran `PYTHONPATH=/data/users/durham/fb-hgext python ../hg/hg --pager=off debuggentrees --verify master~xxxx master` to generate 6 months
worth of trees. It found bugs which are fixed in other patches.
Reviewers: #mercurial
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4724502
Summary:
It'd be nice to have dirsync config in the repo, since it's better synced, and
easier to be modified.
Test Plan: Added a test case
Reviewers: #sourcecontrol, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4732054
Signature: t1:4732054:1489793650:ca6c63ef47a1d9ebd181fbaee5554975801324eb
Summary:
Upstream converted all array.array usages to bytearray, and in doing so they
also changed mdiff.textdiff(array1, array2) to be
mdiff.textdiff(util.buffer(array1), util.buffer(array2)), which we did not do in
our internal implementation.
This fixes that.
Test Plan: Adds a test
Reviewers: #mercurial, quark, simonfar
Reviewed By: quark, simonfar
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4724350
Signature: t1:4724350:1489697253:8f63350bfc30b3079bf38410054b4763942a82f6
Summary:
On macos sshpeer flushes remote output later and it causes tests to fail. Let's
add a call to cleanup() to make sure the output is flushed.
This diff also makes remotefilelog test more robust.
Test Plan: Run tests on macos
Reviewers: #mercurial, tja, mitrandir
Reviewed By: mitrandir
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4721276
Tasks: 15389402
Signature: t1:4721276:1489669109:4ce59f4a1d224d57dbb7c1eb341c4e6a659d2e8c
Summary:
By default debugcheckbackup checks only one backup for the user.
With --all option it checks all backups for the user from all host/reporoots
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4700993
Tasks: 15389402
Signature: t1:4700993:1489548921:800d08f420acc8ef4f807ffd17b31837dbb3fe82
Summary:
Backing up very big repo can cause problems. And very often old commits are
not necessary at all. Let's add a config option to limit the number of heads
that are being backed up.
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4700965
Tasks: 15389402
Signature: t1:4700965:1489549163:4e2c121f01debd7b495486a1f3b062926873399d
Summary:
Previously we saved bookmark hash and revision number as state of
the last backup. Storing last backed up revision is error-prone. I saw
a few corrupted backups where bookmarks pointed to non-backed up
nodes. Also this approach puts more pressure on mysql bundle index.
On every backup it first deletes all existing backup bookmarks and then it
writes new bookmarks. Even if user changed one head or one bookmark
(and that is the usual case) current approach still deletes bookmarks first
and then rewrites them.
Instead let's store all backed up heads and bookmarks locally in json format.
Json was chosen because it's simple and human-readable. Then during backup
let's compare last backed up state to the current state of the repo, and send
to the server only the difference.
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4700930
Tasks: 15389402
Signature: t1:4700930:1489613249:a34369bf53e0718c22258304493dfa27b578857f
Summary:
Let's rename _getbackupstate() to _downloadbackupstate() because it actually
downloads it from the server. Also return class instead of tuple
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4713633
Tasks: 15389402
Signature: t1:4713633:1489612870:3439a6a99ef311bf96784b3b1e53d7e9b94c0713
Summary:
This adds ui.log() output for prefetch statistics. Extensions who hook into
ui.log() can now log this data to external metrics systems.
Test Plan:
Ran a hg prefetch with the config flags enabled, while ptailing the dev command
timer. Verified the result contained remotefilelogfetches*
```
CHGDISABLE=1 FB_HG_DIAGS=1 hg --config
extensions.remotefilelog=../fb-hgext/remotefilelog/ --config
sampling.key.remotefilelog.prefetch=perfpipe_dev_command_timers prefetch -r .~9
ptail -f perfpipe_dev_command_timers | grep durham
```
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4711096
Signature: t1:4711096:1489591144:1c91a4fbd118a3c10c2a2c68391c9f5b0dbcedf3
Summary:
We have encountered a kernel issue where `rmdir` a non-empty directory may race
with other things and hang in kernel for a long time.
This patch changes `os.rmdir` to avoid `rmdir` non-empty directories. It is
written in Cython calling the low-level `readdir` libc friends to make overhead
minimal.
Test Plan: Added a new test
Reviewers: #sourcecontrol, clm, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, simpkins, osandov, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4716711
Tasks: 16647532
Signature: t1:4716711:1489627923:7c7432748c1fd8c070ce257bd172feebd3807f65
test-check-code doesn't like sed -i, so let's switch to a tiny batch script.
Also fix the test importing the extension directly instead of relying on the
path to be right.
