Summary:
Previously, the remote blobstore could be either `git-lfs` or `dummy`. The
application code does not really care, it only wants a "remote" blobstore.
This diff adds a factory method and makes `git-lfs` and `dummy` stores private.
The `@staticmethod get(vfs)` interface is also removed as it's duplicated and
unnecessary - as long as mercurial calls `reposetup`, the blob store objects
are set, and they cannot be missing.
The error message about an unsupported store is also changed to be consistent
with mercurial style. A test was added to test the error.
Test Plan: `rt test-lfs.t`
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4813023
Signature: t1:4813023:1491382755:b9d8ec6518141d0ba8263e16c53f430ce80c39f0
Summary:
It's a good way to check if backup is consistent. If backup is not consistent
then we set special exit code that will be logged. It let's us easily find users
with inconsistent backups and fix it.
Test Plan: arc unit
Reviewers: #mercurial
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4810442
Tasks: 15389402
Summary:
Previously we backed up extinct node if bookmark pointed to them. But this not
is not present in the bundle because `findcommonoutgoing()` intentionally skips
extinct and secret commits. In this case we'll have infinitepush bookmark that
points to non-existent infinitepush commit. To fix it let's filter extinct heads
Test Plan: arc unit
Reviewers: #mercurial
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4803322
Tasks: 15389402
Summary:
The bypass option limits lfs's functionality to only skip hash checks. It is
intended to be used server-side, to make it more predictable - the server
never interacts with the lfs blob service.
Test Plan: Added a test case
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4811828
Signature: t1:4811828:1491301114:c2e3c4200ce4cc84b9c5872a8b9a040176bb002a
Summary:
fbhistedit changes was reverted upstream, revert it here too.
remove username from the test
Test Plan: arc unit
Reviewers: #sourcecontrol, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4830371
Signature: t1:4830371:1491343205:9d7aebd603136fb90ccd3199df60c3e0a5772428
Summary:
Previously errorredirect swallows errors in some cases. This diff makes it
more robust.
- Always log commandexception. `blackbox.log` will probably catch it.
- Print the trace if the handler script cannot be executed.
- Print the trace if the handler script exits with non-zero.
- Add exception for the Ctrl+C case, to not pollute user's terminal.
Test Plan: Added some test cases
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4824767
Signature: t1:4824767:1491299807:3fd863cb99ce71a6c2b59643b0c89eece7985d4b
Summary:
7687edc2cef224a3a5ad4961e11073aa2ea73af8 added HG_HOOKNAME and HG_HOOKTYPE info
Let's update the test
Test Plan: arc unit
Reviewers: #sourcecontrol, mitrandir
Reviewed By: mitrandir
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4826767
Signature: t1:4826767:1491319207:a1c9e9af20cc237ff02bedcb9072f2c0816a1fbe
Test Plan:
arc unit
Note: making sure that int is passed in repo.lookup(...) is not easy.
For example `hg log -r 1` does not pass an int value to repo.lookup().
But `hg bundle` actually does it so use `hg bundle` to verify that fix works.
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4825996
Signature: t1:4825996:1491292671:9bd0897205dff93cb6dee75498827b3826f894da
Summary:
The ideal interactive mode couldn't be implemented trivially. Move `absorb`
to a directory so we can add related, but decoupled components as separate
files.
Test Plan: `make local`
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4813909
Signature: t1:4813909:1491211561:e9c40b1242c9b74230c0b8937723a2d4548e22c3
Summary:
@simpkins fixed up some of the tech debt accumulated in our
Phabricator interfaces, which results in a minor change to output. Most tests
are fixed, but this one got missed.
Fix the test to match his fixes.
Test Plan: Run the fixed test locally
Reviewers: #sourcecontrol, stash
Reviewed By: stash
Subscribers: mjpieters, simpkins
Differential Revision: https://phabricator.intern.facebook.com/D4809914
Signature: t1:4809914:1490955858:bd2c942b9744a5529e8f4fd87bfb50666b53ae6a
Blame Revision: HGEXT8a071fbd26ed6fabf443953183c507606b027bbc
Summary:
This diff makes the `test-lfs.t` much more stronger. It reveals a lot of core
hg issues in this area. I'll send patches to fix them all.
Regarding on lfs, there are some changes:
- An existence check in its push hook was added. Otherwise pushing a revision
with rename will cause crash.
- The "read" processor is responsible for downloading blobs, and translate
raw revision to lfs text. It should always return lfs text. But it may
return raw revision text on error currently. That error handler was
removed to avoid further damage.
Test Plan: Added new test cases. I also added `hg verify` to sanity check things are good.
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4792641
Signature: t1:4792641:1490868946:8f09c84dc9ebda3889d6a1ea04c49a06acbf38a3
Summary:
If the user does not have necesary credentials defined in their arc
configuration, catch the KeyError and convert it into an ArcConfigError.
The existing call sites in the phabstatus and arcdiff extensions catch and
handle ArcConfigError, but not generic KeyErrors.
This also fixes the phabstatus warning messages to end with a newline.
Test Plan: Added a unit test.
Reviewers: #sourcecontrol, quark, simonfar, wez, rmcelroy
Reviewed By: wez, rmcelroy
Subscribers: rmcelroy, net-systems-diffs@fb.com, yogeshwer, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4800977
Tasks: 17002914
Signature: t1:4800977:1490847078:e18bba042e3ff57100e0a7b25c610b5cad17fa2e
Summary:
The traceback changed due to an upstream change, but it's still doing
what we expect.
Change the test to be less sensitive to code changes.
Test Plan: `../../hg-crew/tests/run-tests.py` locally.
Reviewers: #mercurial, quark, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4802376
Signature: t1:4802376:1490872606:8c9b357de7bd65a33abf1d5a3b0568751210aad4
Summary: Let's add a table that will store metadata about infinitepush nodes.
Test Plan: arc unit
Reviewers: #mercurial
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4778867
Tasks: 16899722
Summary:
I forgot to insert it, fixing it now. It didn't cause problems before because
we delete bookmarks only during backups and backups usually have different
bookmark names
Test Plan: arc unit
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4795310
Tasks: 15389402
Signature: t1:4795310:1490810441:0a044ebe776c65b03c633017c33771d32bfa3e77
Summary:
Neither gitnode() template nor gitnode() revset worked on the server. They
both called `hg.peer(repo, {}, 'default)` and since 'default' is not a valid
url these calls failed. To fix it let's wrap `localrepo.lookup()` and call
local repo instead of peer repo if we can't find default path.
Test Plan: arc unit
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4794488
Tasks: 16848812
Signature: t1:4794488:1490795600:24269507f24613883406f0c1838fce302f86df58
Summary:
Previously we don't set findcommonincoming to True, and that bug was unnoticed
because we send this info anyway in server-side `getbundlechunks()` function.
But next diff in the stack uses a fast path server-side which won't work if
fincommonincoming is not set to True.
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4770830
Tasks: 15389402
Signature: t1:4770830:1490719209:c251002d992e244580b7dbfeca0c30dff95734d2
Summary:
Command to list available backups for the user.
--json will be used by automation
Test Plan: arc unit
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4728729
Tasks: 15389402
Signature: t1:4728729:1490005762:41b9683cb7dfc9d84ae1032c807d1f0c3fe60dbf
Summary:
Only match the "Differential Revision" label at the start of a line. We
have some diffs that include legitimate-looking Differential Revision label
strings inside parts of their test plan, which previously confused the phabdiff
output.
Test Plan: Included a unit test.
Reviewers: #sourcecontrol, quark, akushner
Reviewed By: akushner
Subscribers: net-systems-diffs@fb.com, yogeshwer, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4793483
Signature: t1:4793483:1490756514:05b2c49d5d65e1a3eadd5eb78ed0b91ef3bea42c
Summary:
From diff:
```
This test does two things:
1/ Simulate a common condition of pushrebase under load. Normally pushrebase
caches data before acquiring the write lock (if lazy locking is enabled).
Under load, however, when a push has to wait for the lock more often than not,
much of this data becomes out of date and must be refetched once the lock is
acquired. This test simulates that particular case.
Specifically, we create two clients, client1 and client2, both with
nonconflicting changesets to push. client1's push is artificially blocked by a
`prepushrebase` hook (post-caching, pre-lock) that is only released after
client2's push succeeds.
2/ Checks how often we call manifest.read() inside the lock (and outside).
This way we can prevent regressions on manifest reads and test improvements.
manifest.read() is wrapped by an extension that prints a short trace. read calls
inside the lock are marked with a ":(".
```
This test is a bit of a doozy, simplification suggestions are encouraged.
Test Plan: this is, in fact, a test
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: quark, rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4785464
Signature: t1:4785464:1490737637:79d77f6b2e49a428d63b8e1b13d769c82149ee72
Summary:
Previously `lfs.blobstore` must be a relative path. If an absolute
blobstore path is set, it will traceback because vfs audit fails:
```
File "hg/mercurial/revlog.py", line 1356, in _processflags
text, vhash = writetransform(self, text)
File "fb-hgext/hgext3rd/lfs/wrapper.py", line 69, in writetostore
blobstore.local.get(self.opener).write(storeid, chunk)
File "fb-hgext/hgext3rd/lfs/blobstore.py", line 41, in write
fp = self._opener(self.filename(storeid), 'w+', atomictemp=True)
File "hg/mercurial/vfs.py", line 344, in __call__
self.audit(path)
File "hg/mercurial/pathutil.py", line 64, in __call__
raise error.Abort(_("path contains illegal component: %s") % path)
Abort: path contains illegal component: /home/quark/lfslocalblobstore/d7/dbc611df1fe7dfacfe267a2bfd32ba8fc27ad16aa72af7e6c553a120b92f18
```
That was because the code was using `repo.vfs`. This diff adds a new `lfsvfs`
to avoid the issue. The `lfsvfs` also did the correct filename check (the
old `re.match` check will not match the whole string), so `blobstore.local`
could be simplified a lot.
Test Plan:
A new test case was added to make sure absolute blobstore path works. I also
did some cleanups for the test file to de-dup hgrc, and avoid writing files
outside `$TESTTMP`.
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters, remi
Differential Revision: https://phabricator.intern.facebook.com/D4785084
Signature: t1:4785084:1490693306:b42eef8e53af107897c2e1bc2984c090bdd2e465
Summary:
It's the upstream mutable-history that handles the "--fold" case now.
It doesn't have the fix that avoids touching wd in that case.
Test Plan:
tests are passing when mutable-history is updated to default-upstream-tip
I'll delete the actual mutable-history hotfixes once this is accepted.
Reviewers: #sourcecontrol
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4779366
Summary:
A bunch of modifications to get the test pass with the new lfs code.
- Move `lfs` to `hgext3rd`. The code was supposed for hg-core. For now, we do
them in fb-hgext to speed up the process
- Remove the windows test, which is not supported by `run-tests.py` and is
duplicated with `test-lfs.t`.
- Do import `mercurial.i18n._` correctly.
- Change some i18n logic a bit so it's more translator-friendly.
- Change `revlog.RevlogError` to `error.RevlogError`.
- Avoid direct symbol import of `mercurial.util.bytecount`, which will fail the
upstream importchecker test.
- Fix various lint issues like lines being too long etc.
- Document lfs config options.
Test Plan: `arc unit`
Reviewers: #sourcecontrol, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4772216
Signature: t1:4772216:1490401458:1ad3c18ab80e1d31085d0b6b4c630e62a7dc7930
Summary: test-check-code.t now dislikes our docstrings. Fix them up
Test Plan: Run the test locally on my devserver
Reviewers: #sourcecontrol, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4773908
Signature: t1:4773908:1490394131:9d9e5f85b4243119a3615044605bcc5abdecbe4f
Summary: Just as in D4762189. Fixes to match upstream changes
Test Plan: arc unit
Reviewers: #sourcecontrol, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4762596
Signature: t1:4762596:1490286227:004ed0a7055aab64769893e80a995975b62d5ea0
Summary:
changeset: 06d6fecfdb713e3a2e2013de046af0bce4005b96 (@)
user: Jun Wu <quark@fb.com>
date: Mon, 13 Mar 2017 22:19:06 -0700
histedit: use safecleanupnode in _finishhistedit
This simplifies code a lot.
causes significant changes to the test output. Update to match
Test Plan: Run tests locally
Reviewers: #sourcecontrol, quark, stash
Reviewed By: stash
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4762189
Signature: t1:4762189:1490281032:6a48429f60584c4318bd1e247ce2eeb2775d623c
Upstream added __bool__ to manifestdict, and the test checking that
hybridmanifest matches the functions of manifestdict started failing. Let's add
it.
The check-code.py change matches the one in upstream to support the addition of
__bool__.
Summary:
It will allow sparse to not ignore the file, and is more consistent with
other special files.
Test Plan: Updated existing tests
Reviewers: rmcelroy, #sourcecontrol, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4738766
Signature: t1:4738766:1490032493:199f3fef9c74a137b16ae7637b87de625ca5115d
Summary:
This diff just restores the code that was reverted before. Next diffs will fix
it
Test Plan: arc unit
Reviewers: #mercurial, mjpieters
Reviewed By: mjpieters
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4714397
Tasks: 16558840
Signature: t1:4714397:1489717436:9777d783d1443dea46460450e8f3da862c147483
Summary:
This code that reused deltas if the delta parent wasn't available was bugged
because it meant you could end up with a cycle in the delta chains. This was an
old optimization from before trees had history, so let's drop the optimization
(since trees now have history and can be correctly repacked).
Test Plan:
Ran repack on a packfile that previously caused cycles. Verified the
new version did not with `hg debugdatapack foo.datapack'
Reviewers: #mercurial
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4724520
Summary:
It'd be nice to have dirsync config in the repo, since it's better synced, and
easier to be modified.
Test Plan: Added a test case
Reviewers: #sourcecontrol, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4732054
Signature: t1:4732054:1489793650:ca6c63ef47a1d9ebd181fbaee5554975801324eb
Summary:
Upstream converted all array.array usages to bytearray, and in doing so they
also changed mdiff.textdiff(array1, array2) to be
mdiff.textdiff(util.buffer(array1), util.buffer(array2)), which we did not do in
our internal implementation.
This fixes that.
Test Plan: Adds a test
Reviewers: #mercurial, quark, simonfar
Reviewed By: quark, simonfar
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4724350
Signature: t1:4724350:1489697253:8f63350bfc30b3079bf38410054b4763942a82f6
Summary:
On macos sshpeer flushes remote output later and it causes tests to fail. Let's
add a call to cleanup() to make sure the output is flushed.
This diff also makes remotefilelog test more robust.
Test Plan: Run tests on macos
Reviewers: #mercurial, tja, mitrandir
Reviewed By: mitrandir
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4721276
Tasks: 15389402
Signature: t1:4721276:1489669109:4ce59f4a1d224d57dbb7c1eb341c4e6a659d2e8c
Summary:
By default debugcheckbackup checks only one backup for the user.
With --all option it checks all backups for the user from all host/reporoots
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4700993
Tasks: 15389402
Signature: t1:4700993:1489548921:800d08f420acc8ef4f807ffd17b31837dbb3fe82
Summary:
Backing up very big repo can cause problems. And very often old commits are
not necessary at all. Let's add a config option to limit the number of heads
that are being backed up.
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4700965
Tasks: 15389402
Signature: t1:4700965:1489549163:4e2c121f01debd7b495486a1f3b062926873399d
Summary:
Previously we saved bookmark hash and revision number as state of
the last backup. Storing last backed up revision is error-prone. I saw
a few corrupted backups where bookmarks pointed to non-backed up
nodes. Also this approach puts more pressure on mysql bundle index.
On every backup it first deletes all existing backup bookmarks and then it
writes new bookmarks. Even if user changed one head or one bookmark
(and that is the usual case) current approach still deletes bookmarks first
and then rewrites them.
Instead let's store all backed up heads and bookmarks locally in json format.
Json was chosen because it's simple and human-readable. Then during backup
let's compare last backed up state to the current state of the repo, and send
to the server only the difference.
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4700930
Tasks: 15389402
Signature: t1:4700930:1489613249:a34369bf53e0718c22258304493dfa27b578857f
Summary:
We have encountered a kernel issue where `rmdir` a non-empty directory may race
with other things and hang in kernel for a long time.
This patch changes `os.rmdir` to avoid `rmdir` non-empty directories. It is
written in Cython calling the low-level `readdir` libc friends to make overhead
minimal.
Test Plan: Added a new test
Reviewers: #sourcecontrol, clm, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, simpkins, osandov, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4716711
Tasks: 16647532
Signature: t1:4716711:1489627923:7c7432748c1fd8c070ce257bd172feebd3807f65
test-check-code doesn't like sed -i, so let's switch to a tiny batch script.
Also fix the test importing the extension directly instead of relying on the
path to be right.
Summary:
It's just copy-paste from my mutable-history patch series that ended up as
persistent hotfix.
The wrapping logic is very simple: it redirects '--fold' calls to original
metaedit and handles all the rest.
Test Plan:
Included a big copy-pase from test-evolve.t preserving only cases that were
executing metaedit and modified for the smaller test repo (the metaedit tests
were at the end of evolve test file).
Reviewers: #sourcecontrol, quark, simonfar
Reviewed By: quark, simonfar
Subscribers: rmcelroy, quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4708017
Signature: t1:4708017:1489524494:d682aab857e8422bb1a23adc22dddfb9f505b6b3
Summary: New logging will have rotation and will separate different users and repos
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4689191
Tasks: 16474976
Signature: t1:4689191:1489172091:d304d4b19ae6cf52f86c11f442fa760b50d1fdf9
Summary:
For now these command will be used in test because
`wait_for_background_backup.py` tied to the backup state file. And every change
to this file requires change to wait_for_background_backup.py. Let's create
simple internal backup command.
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4667056
Tasks: 15389402
Signature: t1:4667056:1489024344:6c43241e42b7c418baa4e1542f303239ba887c45
Summary:
Upstream has added a matcher arg to manifest.filesnotin, so let's support it in
our native treemanifest implementation.
Test Plan: Added unit tests for filesnotin with and without matchers.
Reviewers: #mercurial, stash
Reviewed By: stash
Subscribers: stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4677030
Signature: t1:4677030:1489076599:23ab67d85bb4fa689f05349253a4d2c4e7d496cc
