Commit Graph

1249 Commits

Author SHA1 Message Date
Jun Wu
0e2c18e2cd remotefilelog: add revlog flags information to the protocol
Summary:
Make the unpacked file format to include the revlog flag information, and make
the getfile(s) protocol support it.

Note: The `getpackv1` protocol and packfile format is not changed yet.

Test Plan:
Run existing tests. Stronger tests and lfs integration test will be added when
this area is more complete.

Reviewers: #mercurial, rmcelroy, durham

Reviewed By: durham

Subscribers: rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4903772

Signature: t1:4903772:1493152451:ab393b0208f0eee199ffc4c8fcfdfd5dd6d0f3ac
2017-04-26 13:08:13 -07:00
Stanislau Hlebik
64ecdf2691 fastpartialmatch: handle nullid correctly
Summary: Special case for nullid

Test Plan: arc unit

Reviewers: #mercurial, simpkins, ikostia

Reviewed By: ikostia

Subscribers: net-systems-diffs@fb.com, mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4945927

Tasks: 17622253

Signature: t1:4945927:1493123209:712990c1e883241d99c0aeb4af8d48e18e1baa3d
2017-04-26 03:55:59 -07:00
Jun Wu
6c6ebce1f6 morecolors: new extension to add more colors to outputs
Summary:
The extension was a quick hack from me to make it easier to locate non-core
code when doing extension development. It seems a good idea to add colors in
general so it's named "morecolors" with the hope that we can colorize more
places in the future.

Test Plan: Added a test

Reviewers: #mercurial, durham

Reviewed By: durham

Subscribers: durham, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4884999

Signature: t1:4884999:1492473666:89a6756b231eac4274fa05d489480229146bdab8
2017-04-25 17:03:26 -07:00
Jun Wu
76c7347a2e clib: import sha1 impl from github/cr-marcstevens/sha1collisiondetection
Summary:
This imports a SHA1 hash function implementation from
https://github.com/cr-marcstevens/sha1collisiondetection, commit hash
0572d8a302b1f62f25c1559b0de74908550ab2c3, using the MIT License.

This allows us to decouple from an external crypto library like openssl. The
new library's "safe hash" feature could also mitigate known SHA1 collisions.

test-check-code is modified accordingly to skip checking the sha1 code
without modifying them.

Test Plan: Run test-check-code.t and make sure it's not complaining.

Reviewers: #sourcecontrol, durham

Reviewed By: durham

Subscribers: ikostia, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4945012

Signature: t1:4945012:1493154822:be0deb451d50f3a8b6cc42addf091349bf6554f4
2017-04-25 14:53:04 -07:00
Jun Wu
bedfb2642c check-code: remove check-code.py
Summary:
We now use `check-code.py` directly from core mercurial. It's no longer
necessary to have it in our repo.

Test Plan: `arc unit`

Reviewers: #sourcecontrol, jeroenv, rmcelroy

Reviewed By: jeroenv, rmcelroy

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4945054

Signature: t1:4945054:1493122097:5880dc2c0caac215cb10db40d30691dc1a147315
2017-04-25 14:47:17 -07:00
Jun Wu
44c4a80415 pushrebase: deal with phase move correctly
Summary:
Previously we ignore "draft -> public" phase move, and new nodes returned by
the server may remain "draft" until the next pull. This diff records the
obsmarkers and updates phases of the new nodes correctly so they become
public without an extra hg pull.

Test Plan: Modified an existing testcase

Reviewers: #mercurial, durham, rmcelroy

Reviewed By: durham

Subscribers: durham, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4934720

Signature: t1:4934720:1493076241:9b23da1140b68aece9e838c604e64611b3498794
2017-04-25 14:46:36 -07:00
Phil Cohen
eb2814985f resolve: add internal:dumpjson as an internal merge tool (as an extension, for now)
Summary:
This basically takes the last upstream version of my conflictinfo patch and makes it an extension, so we can get it out to FB/Nuclide users
a bit faster. @asriram has already been developing with this version as a personal extension for a couple of weeks.

I still plan to ship the upstream version through after the freeze ends -- when that happens, we can delete this extension. During the time the two
versions overlap, they shouldn't conflict. (The extension version will still run over the internal version until it's disabled, though.)

Review-wise, this is similar to the last upstream version, except it adds the "command" key that indicates which command generated the conflicts (based on which mutually-exclusive state file exists), and incorporates most of the feedback.

Test Plan: Added a test and ran associated tests.

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, asriram, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4944709

Signature: t1:4944709:1493148790:a4e798f5bd17ada767ae6c96fe8c8ab973960383
2017-04-25 12:46:28 -07:00
Greg Prosser
094b505444 scm-prompt: fix _dotfiles_scm_info
Summary: I was using """_dotfiles_scm_info '[%s] '""" in my prompt, which broke after the refactoring.  Add a unit test and fix the code.

Test Plan: wrote a test for the breakage, edited code until they all passed

Reviewers: rmcelroy, tja

Reviewed By: rmcelroy

Subscribers: mjpieters, #pe_security

Differential Revision: https://phabricator.intern.facebook.com/D4945132

Signature: t1:4945132:1493136425:4516151982d173584cf874985de9de1fed0962d5
2017-04-25 10:56:35 -07:00
David Soria Parra
63370254c8 p4fastimport: implement --bookmark
Summary:
Implement an option to set a bookmark after we imported successfully.
We used to try to calculate this for continuous imports in a wrapper around the
importer. However it's much easier to just do it inside the importer itself, in
particular when we add branch support later

Test Plan: rt test-p4*

Reviewers: #idi, #sourcecontrol, quark

Reviewed By: quark

Subscribers: quark, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4935717

Signature: t1:4935717:1493127835:262955b3288d9bd03ca08a45d7ec1667d786430a
2017-04-25 15:29:39 +01:00
David Soria Parra
0337eae344 p4fastimport: correct PYTHONPATH for tests
Summary:
correct PYTHONPATH to allow running tests without having to
source hg-dev. This was suggested by @durham

Test Plan: rt tests-p4*

Reviewers: #sourcecontrol, #idi, mjpieters

Reviewed By: mjpieters

Subscribers: mjpieters, durham

Differential Revision: https://phabricator.intern.facebook.com/D4921199

Signature: t1:4921199:1492696261:ca70d837a9f273433b21fb10e3f4ff1afc6a5698
2017-04-25 15:29:39 +01:00
David Soria Parra
78dcff9f00 p4fastimport: support client view
Summary:
Perforce client support a view. A view maps a server side path to a client side
path, e.g.: the view '//depot/A/B/... //myclient/foo/...' maps every file in
//depot/A/B/ on the server side to a local checkout foo/ inside the root for
checkouts defined for the client 'myclient'.
We are using §p4 where§ to support these mappings. We do the mapping inside the
FileImporter at the moment as this runs nicely in parallel. It's a bit hacky
but get's the job done. We use this mostly to ommit the common prefix
//depot/... and remove branch indicators such as Main.

So in our case a view looks like

  //depot/Software/OculusSDK/PC/Main... //client/Software/OculusSDK/PC/...

resulting in a file

  //depot/Software/OculusSDK/PC/Main/test.txt

being imported as

  Software/OculusSDK/PC/test.xt

Test Plan: rt test-p4*

Reviewers: #sourcecontrol, #idi, ikostia

Reviewed By: ikostia

Subscribers: ikostia, durham, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4913483

Signature: t1:4913483:1492702356:b97b691343b8a1d52940445934730b31d411db4c
2017-04-25 15:29:39 +01:00
David Soria Parra
2ab5a0baf8 p4fastimport: support for writing LFS metadata to sqlite
Summary:
We do not write the blobs to local cache anymore. We want our LFS
server to import them from Perforce directly or serve them from Perforce
directly. In order to do so, we need the correct mapping from oid to perforce
file + cl. This is generally useful metainformation that other LFS
implementation can use. We simple write the data to sqlite because it's simple
and built in.

Test Plan: rt test-p4*

Reviewers: #sourcecontrol, #idi, quark

Reviewed By: quark

Subscribers: quark, durham, wlis, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4913469

Signature: t1:4913469:1492796253:1e3b389c7cb0ba3acf9504410d267a1cf9651118
2017-04-25 15:29:39 +01:00
David Soria Parra
7d78fa39bf p4fastimport: initial support for writing lfs metadata
Summary:
Add a special mode to the importer that patches the LFS extension to
not write blobs to local disc. In our case we do have the files already in
Perforce and do not have to write them again to disk. This is currently breaking
verify and therefore we are patching verify.

Test Plan: rt test-p4*

Reviewers: #sourcecontrol, #idi, durham, quark

Reviewed By: quark

Subscribers: quark, durham, wlis, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4913455

Signature: t1:4913455:1492979419:204c1075376fe975ddea880b22e6984684e7ff25
2017-04-25 15:29:39 +01:00
Stanislau Hlebik
1642b009fa infinitepush: add isbackedup command
Summary:
Command that checks if a commit was backed up.
It does it by checking if revisions are in local backup state.

Test Plan: Run unit tests and lints

Reviewers: #mercurial, durham

Reviewed By: durham

Subscribers: durham, mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4938072

Tasks: 17519836

Signature: t1:4938072:1493072854:291765bb59d327db8504feb47d6089818ae1e11a
2017-04-25 01:09:28 -07:00
Stanislau Hlebik
d610259241 infinitepush: delete bkpstate from shared working copy
Summary:
infinitepushbackup.tempcleanworkingcopiesbackups option sends special command
to clean backup bookmarks from the server for the shared working copy (i.e.
if we have main repo `fbsource` and shared working copy `fbsource2` we want
to have backups only for `fbsource` not for `fbsource` and `fbsource2`).
Before this diff cleanup commands were send for every backup.
This diff makes them send only if backup state file is present.

Test Plan: arc unit

Reviewers: #mercurial, durham

Reviewed By: durham

Subscribers: durham, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4929489

Signature: t1:4929489:1493075761:a408300338a10864043b538540d03880a49c4e1a
2017-04-25 01:09:28 -07:00
Rodrigo Damazio Bovendorp
fe81fd7cdc backups: small output improvements for backups extension
This allows us to turn off the obsolescence warning altogether, since we still
use strip together with obsolescence in some situations (e.g. changing the
narrowspec for narrowhg, in which case we "insert" revisions which were
previously not relevant in the middle of the change log and then have to
strip and re-create their children to repoint p1).
2017-04-24 10:00:31 -07:00
Stanislau Hlebik
fceac7d85b phrevset: return baseset instead of list
Summary:
There was a recent change in mercurial and revsets
shouldn't return lists anymore. This diff fixes it.

This diff also fixes git diff handling. It returned not even a list,
but a single rev number. Unfortunately it's not tested because we need to
mock conduit response.

Test Plan: arc unit

Reviewers: #mercurial, ikostia

Reviewed By: ikostia

Subscribers: ikostia, kulshrax, mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4938930

Signature: t1:4938930:1493051696:cfb278773e5932d5f38942b0a0ca49f018ec5083
2017-04-24 09:47:52 -07:00
Stanislau Hlebik
c15b2d9958 infinitepush: read backup state file from real repo not from shared working copy
Test Plan: arc unit

Reviewers: #mercurial, ikostia

Reviewed By: ikostia

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4929323

Signature: t1:4929323:1492785417:609a5569ad49f1f9e938908c30214a7b0a89ffb6
2017-04-24 01:07:20 -07:00
Jun Wu
00a9465989 lfs: be compatible with filelog metadata
Summary:
This diff changes lfs `revision(raw=False)` output to include hg filelog
metadata. The LFS blob does not contain filelog metadata as before.

This hurts performance if there is a rename, or the binary starts with the
magic `\1\n`. But compatibility is greatly improved - it's now possible to swap
a non-lfs revision with mercurial rename to a lfs revision, and easier to be
compatible with remotefilelog (namely, remotefilelog defers filelog.add until
commit hash is known).

Test Plan: Modified existing test.

Reviewers: #mercurial, durham

Reviewed By: durham

Subscribers: rmcelroy, durham, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4906074

Signature: t1:4906074:1492559513:09b25fc1026d4ce8fd784a044d6724f12e8bda45
2017-04-21 19:56:27 -07:00
Stanislau Hlebik
b6eb201c3f infinitepush: fix backup of commits with removed files
Summary:
There was a bug that caused commits with file deletions to not be backed up.
This diff fixes it by first checking if file exists in the commit and only
then downloading context of the file.

Note:
In tests I had to ignore stdout of `hg pushbackup` because the output was
different on macs and linux.
On linux there was an additional line

  remote: abort: data/committostripfirst.i@091b63e5e4: no match found!

Probably mac's remotefilelog closes stdout/stderr earlier, but I wasn't
able to find a root cause.

Test Plan: Run unit tests

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: quark, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4929244

Tasks: 15389402

Signature: t1:4929244:1492791804:77b2baa9eb54a53120a955e72e6c132be5db6b44
2017-04-21 10:24:14 -07:00
Kostia Balytskyi
822a366b5b tests_: fix osx compatibility in ls output
Summary:
BSD and GNU ls behave differently in case of a missing dir:
- exit code is [1] vs [2]
- message is also slightly different
We want this to be unified for both platforms.

Test Plan:
- run the test on Linux
- hope for the best on OSX

Reviewers: #sourcecontrol

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4921369
2017-04-20 05:46:20 -07:00
David Soria Parra
44a0aabf25 p4fastimport: incremental imports
Summary:
Implement incremental imports.

1. Find the last imported perforce changelist.
2. Set startctx and use it in all importers
3. Import filelogs from their current position (we "should" add an additional check here, but we don't)
4. Import manifests and changelists.

Manifests are a bit tricky because we must obtain the original filelog revision
*BEFORE* we imported them, but manifest imports come "after". We could read the
most recent entry from manifests, but that won't cover the case in which files
are added. So instead we know the changelist that we are currently importing,
and looking for the rev with the correct linkrev in filelogs. That's a big ugly,
but it works. We could instead return the original offset from the worker and
pass it into the manifest importer, but I feel that is not much better and
evenutally more errorp rone.

Test Plan: cd tests && rt test-p4*

Reviewers: #sourcecontrol, durham

Reviewed By: durham

Subscribers: durham, mjpieters, #idi

Differential Revision: https://phabricator.intern.facebook.com/D4890110

Signature: t1:4890110:1492662991:0e141e62734e1224ac8e1c11f4e8794452455b18
2017-04-19 23:33:06 -07:00
David Soria Parra
ad8ee588ed p4fastimport: more detailed debug output
Summary:
More detailed debug output for manifests. For changelog we don't
use it at the moment as changelog imports uses the perforce date, causing
changelog hashes not being stable. Will fix this in the future.

Test Plan: cd tests && rt test-p4*

Reviewers: #sourcecontrol, durham

Reviewed By: durham

Subscribers: durham, mjpieters, #idi

Differential Revision: https://phabricator.intern.facebook.com/D4890093

Signature: t1:4890093:1492474213:2b410da985fdd5d0786ef9dde05ecbb96f157e14
2017-04-19 23:33:06 -07:00
David Soria Parra
ba15cdea75 p4fastimport: use repo lookup to get p1/p2
Summary:
Use repo lookup to get p1 and p2 and use their manifestnodes instead.
We will use this for incremental imports in order to correctly optain the
manifestnode.

Test Plan: cd tests && rt test-p4*

Reviewers: #sourcecontrol, durham

Reviewed By: durham

Subscribers: durham, mjpieters, #idi

Differential Revision: https://phabricator.intern.facebook.com/D4890084

Signature: t1:4890084:1492474086:91658b5c57b4e58af88ee14673ffa3516bc2d88a
2017-04-19 23:33:06 -07:00
Durham Goode
f17ea5b392 treemanifest: add config to not clone tree revlogs during remotefilelog clone
Summary:
Previously a streaming clone would pick up the 00manifesttree.i file. This patch
adds a config to prevent that file (and other meta/ manifests) from being
transfered. The config defaults to False since Google is currently depending on
the previous behavior.

Test Plan: Add a test

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4915481

Signature: t1:4915481:1492641427:6c8a91da4c8c6e00e23f96a93807d3c426b50532
2017-04-19 21:14:04 -07:00
Durham Goode
f1a7d04b25 treemanifest: add test for prefetch with base nodes
Summary: Now that prefetching supports base nodes, let's add a simple test covering it.

Test Plan: Ran the test

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4911537

Signature: t1:4911537:1492641400:752c52acfb264ec42c5d05865bf87ebfc2558801
2017-04-19 21:14:04 -07:00
Durham Goode
2ed79d7eb9 treemanifest: fix test flakiness
Summary:
The old test output had some flakiness where:

1. The linknode for y:1406e7411862 would flicker between commit 0 and commit 6,
even though y never existed in commit 0. This is caused by some ambiguity in how
the store represents the ancestor list (it's keyed on node instead of (name,
node)). To fix the test let's just modify file y so it doesn't have the same
hash as file x at any point.

2. The order of nodes in the histpack would flicker because there was no link
between the two separate histories of file y. This is caused by a lack of
sorting of the separate roots before writing to the histpack.

Test Plan: Ran the test many times

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: quark, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4915472

Signature: t1:4915472:1492641328:d4542665e3a69fe2fc188b450f75ff10e6d681de
2017-04-19 21:14:04 -07:00
Durham Goode
248785548b treemanifest: add support for minimal tree pack sending
Summary:
The gettreepack protocol already had the arguments for the client to specify
what nodes they want and what nodes they have, but it wasn't implemented (we
just always returned the entire tree). This patch adds support for only sending
parts of the tree that are new. In addition, if the client requests many nodes,
we will only send the portion of each node that is not contained in the other
nodes. For instance if you ask for X and Y, where X is the parent of Y, then we
will send all of X and only the part of Y that is new.

The order changed a bit, probably because we stopped over delivering data.

Test Plan:
Ran and updated the tests. The next patch will actually add a test
for this code.

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: quark, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4911531

Signature: t1:4911531:1492643897:58aabe5a32f156dd7424cc07c990301dbd5dc1c7
2017-04-19 21:14:04 -07:00
Durham Goode
2565a8cd22 treemanifest: change test to use multiple directories
Summary:
A future patch will make tree prefetching more efficient. Let's change the
prefetch test to have changes in two separate directories so we can observe that
one directory is not downloaded when doing an efficient prefetch.

Test Plan: Ran the test

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4911523

Signature: t1:4911523:1492641009:ac108ebac9573444c87c403f0be0b246d734dda7
2017-04-19 21:14:04 -07:00
Durham Goode
dae50fc99e treemanifest: support repacking revlogs into packs
Summary:
Accessing treemanifests in revlogs is incredibly slow. This patch adds the
ability for `hg repack` to repack the revlog content into a data pack file, and
teaches the getserverpack code to look in the pack file first.

Test Plan: Need to add a test. Sent out anyway for review

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: quark, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4901298

Signature: t1:4901298:1492640706:2850b9f0e9bbee77952f46af3b784aa81253e626
2017-04-19 21:14:04 -07:00
Durham Goode
42344a282f treemanifest: fix fncache integration
Summary:
Future tests showed that backfilltree was not populating the fncache, which
meant the files were not streamed during a clone or read during a repack. This
fixes this by using the actual repo store (which handles fncache serialization)
instead of a creating a new store.

Test Plan: Added a test

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: quark, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4916070

Signature: t1:4916070:1492647129:4deeb70c63dd414f6d5c06e4259a0bbf899f3425
2017-04-19 21:14:04 -07:00
Durham Goode
10f743a300 treemanifest: chop off trailing slash when requesting manifests
Summary:
The name being passed to the store was wrong, because it had a trailing slash.
This wasn't caught before because the code was reading and writing the paths
with a slash at the end, so it matched as long as we only interacted with packs
produced by the code. The issue became more obvious when I tried to have packs
generated from revlogs interact with this code.

All the tests are affected since the entry keys changed.

Also use 'const ManifestFetcher &x' to pass the ref to avoid the copy.

Test Plan: Tests updated

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: quark, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4901274

Signature: t1:4901274:1492638476:ff28f8976657baec99effbd82ecd436f6282ea5b
2017-04-19 21:14:04 -07:00
Durham Goode
b30215f840 treemanifest: change server to use ctreemanifest for reading revlogs
Summary:
The server treemanifest reading code was super slow because it used the pure
python treemanifest implementation. Let's switch it to use the ctreemanifest
implementation on top of a revlog store class.

The tests change because the output is now in reverse topological order (deepest
trees first).

Test Plan: Tests updated

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4901268

Signature: t1:4901268:1492637601:371a2303ac4f6e2b81f69a85ca657849150abcf7
2017-04-19 21:14:04 -07:00
Durham Goode
45a47ec954 treemanifest: fix serialization during walksubtree of non-finalized trees
Summary:
Previously, walksubtree would segfault if you ran it on a tree that had
non-finalized entries, since they contain node pointers that point to NULL.
Let's fix this by having serialization refer to the nullid if the pointer is
null.

Test Plan: Updated the tests to cover this

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4901257

Signature: t1:4901257:1492637151:803b3183ad766e34f0ad24f14a8ec1b784600879
2017-04-19 21:14:04 -07:00
Durham Goode
c0c997fb89 treemanifest: add walksubtrees api
Summary:
Adds a walksubtrees api that returns a set of subtrees from the given
treemanifest. This will be used in a later patch to return just the subtrees
that are new in one treemanifest. For now it's only used by the unit tests

Test Plan: Added unit test

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: quark, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4901241

Signature: t1:4901241:1492645063:5b3617cd36f3205bef8e09c4d37dd132e0bece4e
2017-04-19 21:14:03 -07:00
Durham Goode
05f39b75e9 treemanifest: add test for no-op subtree change
Summary:
Adds a test for when a tree is modified, then modified back to the original
content. When serializing, it should not result in new trees for that directory,
but should result in a new root tree since p1 has changed.

Test Plan: Ran the test

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: quark, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4880087

Signature: t1:4880087:1492629218:93de40aede28b6b0f5b02e7015d96c61e531cc6a
2017-04-19 21:14:03 -07:00
Durham Goode
f2405cda64 treemanifest: update finalize tests
Summary:
The current tests test some undefined behavior of manifest.finalize() where it
can rewrite immutable Manifest instance's with a new node. This happens if you
call finalize without passing the appropriate parent that the manifest was
copied from. So the new tree contains unmodified subtrees from the original
manifest, and since we don't pass the original manifest to finalize, it doesn't
realize these trees already existed and tries to give them a new node.

A future patch makes tree mutability more strict and caught this. Fixing the
tests now, so I can show that future patches don't break this.

Test Plan: Ran the test

Reviewers: #mercurial, stash

Reviewed By: stash

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4880057

Signature: t1:4880057:1492589057:a7026bc791ea2cd12aa790513cf46d5180cb0638
2017-04-19 21:14:03 -07:00
Durham Goode
8da9d7b37f treemanifest: automatically download missing trees on demand
Summary:
This adds a remote tree datastore to the store, so if we don't have a tree on
local disk, we can reach out to the server for it.

This also makes a small change to fastmanifest to use get() instead of
getmissing() to test if the tree is available. getmissing() would indicate the
tree is missing if it wasn't local, while get() will attempt to fetch it from
the server.

Test Plan: Added a test

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: quark, stash, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4864645

Signature: t1:4864645:1492627823:7afff5db54c895507507e57ab7263f4dbf10b5f7
2017-04-19 21:14:03 -07:00
Durham Goode
b44263e197 fixtests: move treemanifest configs onto client only
Summary:
The treemanifest tests were setting the configs on both the client and the
server, without telling the server it was a server and without setting it up
appropriately to be a client. This caused the tests to fail in a future patch.
Let's move the treemanifest config to the clients, since that's the only bits
being tested here.

Test Plan: Ran the tests

Reviewers: #mercurial, ikostia

Reviewed By: ikostia

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4864766

Signature: t1:4864766:1491903651:ab91eed5a2eb8a60db8e230c2ba5c0ee29b5adc9
2017-04-19 21:14:03 -07:00
Durham Goode
3633627260 treemanifest: add prefetchtrees for receiving trees from the server
Summary:
This adds a new prefetchtrees command and gettrees wireprotocol command for
receiving trees from the server. The actual wire protocol is a custom request
format, with a normal pack file response format (same as used in remotefilelog).
The request args can specify which manifest nodes are wanted, for which
directory and which subdirectories.

I attempted to use the changegroup3 response format instead of the pack format,
but changegroup is extremely intertwined with revlogs and with producing
changegroups for linear swaths of the repo history, instead of random parts of
just the manifest.

Test Plan: Added a test

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: stash, quark, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4864435

Signature: t1:4864435:1492626392:c3603eeb261a1f9b9d836f0d44614025b58d45ea
2017-04-19 21:14:03 -07:00
Arun Kulshreshtha
02e5cc57ba tweakdefaults: allow hg update --merge without --nocheck
Summary: The --merge flag is relatively new; we shouldn't add --check if it is specified.

Test Plan: Unit test

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4916396

Tasks: 17490374

Signature: t1:4916396:1492647440:d9955931cb9177719a58325c08a715d4d51d1226
2017-04-19 17:20:23 -07:00
Arun Kulshreshtha
6065ceef3f fbamend: use unfiltered repo when clearing preamend bookmarks
Summary: Sometimes restack attempts to clear preamend bookmarks on hidden commits. This only happens when the user has a configuration error (typically inhibit is disabled) which would mess up other parts of restack. Nonetheless, there is no reason to crash here.

Test Plan: Disable inhibit, perform an amend, and run `hg restack`. Unit test is forthcoming.

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4856640

Tasks: 16726367

Signature: t1:4856640:1491816803:56beeb11360a7a2f6bc6243069c562b08a015afc
2017-04-19 15:52:45 -07:00
Kostia Balytskyi
aeef0ad5cc compatibility: migrate from scmutil.vfs to mercurial.vfs.vfs
Differential Revision: https://phabricator.intern.facebook.com/D4908906
2017-04-18 14:42:33 -07:00
Stanislau Hlebik
928a96b602 infinitepush: use ui.debug() instead of ui.warn()
Summary: To avoid spamming users

Test Plan: Run tests

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4883447

Signature: t1:4883447:1492088221:3649b14ff514957d2f412aae2647ca1b20ef5fea
2017-04-17 23:44:26 -07:00
David Soria Parra
61c9ea8ec7 p4fastimport: require p4 during tests
Summary: Require perforce during all tests.

Test Plan:
run tests with and without Perforce in PATH. Tests correctly run with
P4D and P4 in path and were correctly skipped without.

Reviewers: #sourcecontrol, #idi, durham

Reviewed By: durham

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4899085

Signature: t1:4899085:1492475858:3bd1443c707e56461835d278a9c6bf3e034b5f4a
2017-04-17 18:08:07 -07:00
Arun Kulshreshtha
a72a47d515 absorb: s/chunks/chunk
Summary: Noticed this minor gramatical error when using absorb.

Test Plan: Unit tests.

Reviewers: quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4887040

Signature: t1:4887040:1492123047:dcae53cabd64f1df13d80a20953c712ce0aaaa09
2017-04-13 16:00:51 -07:00
David Soria Parra
ef08c10f5b p4fastimport : introducing fast Perforce to Mercurial convert extension
Summary:
`p4fastimport` is a fast convert extensions for Perforce to Mercurial. It
is designed to generate filelogs in parallel from Perforce. It tries to
minimize the use of Perforce commands and reads from the the Perforce
store on a Perforce server directly.

The core of p4fastimport is the idea to generate a Mercurial filelog
directly from the underlying Perforce data, as a Perforce file in most
cases matches a filelog directly (per-file branches is an exception). To
generate a filelog we are reading each file for an imported revision. A
file in Perforce is locally either stored in RCS, as a compressed GZIP
or as an flat file (binaries). If we do not find a version locally on
disk we fallback to downloading it from Perforce.

We are generating manifests after all filelogs are imported. A manifest
is constructed by adding and removing files from an initial state. We
are generating the correct offset from a manifest into the filelog by
keeping track of how often a file was touched.

We then generate the changelog.

Linkrev generation is a bit tricky. For every file in Perforce know
to which changelist it belongs, as it's stored revisions contains the
changelist. E.g.  1.1422 is the file changed in the changelist 1422 (this
refers to the "original" changelist, before a potential renumbering,
which is why we use the -O switch).  We use the CL number obtained
from the revision to reverse lookup the offset in the sorted list of
changelists, which corresponds to it's place in the changelog later,
and therefore it's correct linkrev.

Parallel imports: In order to run parallel imports we MUST keep one lock
at a time, even if we import multiple file logs at the same time. However
filelogs use a singular `fncache`, which will be corrupted if we generate
filelogs in parallel. To avoid this, repositories must be generated with
*fncache* disabled! This restricts `p4fastimport` with workers to run
only on case sensitive file systems.

Test Plan:
The included tests as well as multiple imports from a small testing
Perforce client. Afterwards successfully run `hg verify`

  make tests

Reviewers: #idi, quark, durham

Reviewed By: durham

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4776651

Signature: t1:4776651:1492015012:0161c4f45eab4d3b64597d012188c5f2007e8f7d
2017-04-13 11:11:09 -07:00
Stanislau Hlebik
57dc185f42 infinitepush: don't make separate backups for different working copies
Summary:
Previously we had separate backup per working copy. That's very confusing
since all these working copies shares the same repo. This diff fixes it and
also adds config option to clean unnecessary working copy server-side.

Test Plan: Run infinitepush unittest

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: rmcelroy, quark, mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4876230

Signature: t1:4876230:1492025747:3579e5046efc2ed309044fc3335c36ac4f7bdd04
2017-04-13 04:58:11 -07:00
Adam Simpkins
642dea7e2c fixcorrupt: automatically look back as far as necessary
Summary:
Update the fixcorrupt extension to look back as far as necessary in the revlog
to find a good entry.  Previously the user had to supply a good value using the
--checklen argument.  If --checklen was too small, fixcorrupt could end up
trying to truncate in the middle of a corrupted section of the revlog, which
would cause problems as it was still using corrupted data.

This change ensures that fixcorrupt always looks back far enough to find a good
entry.  We start by looking at the last 10 entries, and double that amount each
time the first item in the list is still bad.

Test Plan:
Used it to fix a corrupted fbsource repository where the corruption was 36
commits back.

Reviewers: #mercurial, quark, stash

Reviewed By: stash

Subscribers: stash, net-systems-diffs@fb.com, yogeshwer, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4874243

Signature: t1:4874243:1492007518:7406b03a9967815a496a8c2fae394c5f21f7e60a
2017-04-12 14:09:22 -07:00
Mateusz Kwapich
75778a9b81 remotefilelog: fix tests
Summary: Only one exact form on "hg" invocation with --stdio is now allowed

Test Plan: tests are passing now

Reviewers: #mercurial, simonfar

Reviewed By: simonfar

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4877162
2017-04-12 13:21:16 -07:00
Stanislau Hlebik
8f8cdc2d49 hgext3rd: cleanobsstore extension
Summary: See comments and docstring

Test Plan: arc unit

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: quark, mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4851138

Signature: t1:4851138:1491589247:5ca750e45bcc9fc996c9d7c14e31cb5f24ea983f
2017-04-12 01:16:01 -07:00
Stanislau Hlebik
efbec7652d infinitepush: fix security concern
Summary: Check owner of a log file before writing to it. See comments for details

Test Plan: arc unit

Reviewers: #mercurial, simpkins

Reviewed By: simpkins

Subscribers: net-systems-diffs@fb.com, mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4842933

Tasks: 17155924

Signature: t1:4842933:1491861137:e8a027fb7a930c0c5f553c75cb84214d24f66ce3
2017-04-12 01:12:23 -07:00
Jun Wu
e0eb41504b test-fixcorrupt: be OSX compatible
Summary:
`wc` in OSX has different padding behavior that breaks the test:

```
       $ wc -c .hg/store/00changelog* .hg/store/00manifest* | sort
    -   99 .hg/store/00manifest.d
    -  110 .hg/store/00changelog.d
    -  128 .hg/store/00changelog.i
    -  128 .hg/store/00manifest.i
    -  465 total
    +        99 .hg/store/00manifest.d
    +       110 .hg/store/00changelog.d
    +       128 .hg/store/00changelog.i
    +       128 .hg/store/00manifest.i
    +       465 total
```

This patch fixes it.

Test Plan: The fix was verified manually on an OS X machine.

Reviewers: #mercurial, durham

Reviewed By: durham

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4872689

Signature: t1:4872689:1491954172:92178178deea88ec3545c42ad0cdecc8099a1db6
2017-04-11 16:43:32 -07:00
Jun Wu
340750c4f4 pushrebase: send obsmarkers to client if client supports it
Summary:
Previously when the client has obsstore enabled, and the server has obsstore
disabled, clients won't get the obsmarkers, which is suboptimal. This diff
makes the server send obsmarkers in that case.

Practically, this means people will no longer need to run `strip` after landing
a diff in this repo.

Note: `test-pushrebase-manifests.t` is somehow flaky about the empty lines. I
guess they are not related to the change. But the test change is not 100%
reproducible.

Test Plan: Added a new test case

Reviewers: #mercurial, stash

Reviewed By: stash

Subscribers: rmcelroy, ikostia, stash, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4865150

Signature: t1:4865150:1491895844:afbc3079a40a9a9fa9af1eab4eeaca91091e8d2d
2017-04-11 13:21:27 -07:00
Jun Wu
de5e7c6545 fixcorrupt: new extension to fix corrupted repo
Summary:
We sometimes get reports about corrupted repos. Usually the corrupted part
is just at the end of changelog or manifest.

Truncating them manually works but people need to be very careful. This
extension is like the manual fix but automatized.

Test Plan:
Run `hg debugfixcorrupt --no-dryrun` on a reported corrupted repo and check
it truncates files correctly and the repo passes `hg verify` afterwards.

Reviewers: #mercurial, durham, stash

Reviewed By: stash

Subscribers: stash, rmcelroy, durham, lcharignon, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D3408396

Signature: t1:3408396:1491897232:fc17a105124b568963441adfec97e26735df3258
2017-04-11 13:19:42 -07:00
Adam Simpkins
fdc2c3e6f9 extutil: add unit tests for runbgcommand
Summary:
Add some basic unit tests for extutil.runbgcommand().

This also changes the behavior to throw an OSError if we fail to execute the
process, instead of a subprocess.CalledProcessError.  This matches the behavior
of the subprocess module when it fails to execute the command.

Test Plan: Ran the tests.

Reviewers: stash, quark

Reviewed By: quark

Subscribers: net-systems-diffs@fb.com, yogeshwer, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4864999

Signature: t1:4864999:1491874918:c03edafe02af217e41c28a770137bfd72bcbba9b
2017-04-11 11:25:40 -07:00
Ryan McElroy
8267123b73 scm-prompt: add eden snapshots support
Summary:
Based on the recent commit wez made, but centralized to the scm-prompt
source-of-truth.

Test Plan: updated test

Reviewers: #sourcecontrol, wez, simpkins, simonfar

Reviewed By: simonfar

Subscribers: net-systems-diffs@fb.com, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4867662

Tasks: 17110799

Signature: t1:4867662:1491916786:647edb819a059a3a6baa09c8d189ea706933687b
2017-04-11 08:47:12 -07:00
Jun Wu
46d44c8b07 lfs: use non-chunking spec format if possible
Summary:
The "chunking" feature is not specified by [the current Git LFS standard](21e1695220/docs/spec.md).

Therefore avoid using it if possible - if there is only one chunk, use the
standard specification (`https://git-lfs.github.com/spec/v1`).

An upload message is slightly changed to be more accurate.

Test Plan: Changed existing tests

Reviewers: davidsp, #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4856527

Signature: t1:4856527:1491638822:1c6f555b706e7bb22dd9090afa156f2161bf9f7f
2017-04-10 11:17:48 -07:00
Jun Wu
38632025a9 lfs: add bundle support
Summary:
This diff adds bundle support for lfs:

  - Let `hg bundle` use changegroup3 instead of changegroup2 to record revlog
    flags.
  - Hook related functions so `hg -R bundle.hg` works with LFS.

Test Plan: Added a test

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4856335

Signature: t1:4856335:1491638660:d56ba54ea5f59262f009418b9c7e48c8f2a25ed6
2017-04-10 11:11:37 -07:00
Jun Wu
cdc9accca5 lfs: do not store mercurial filelog metadata in lfs blobs
Summary:
Per discussion with @davidsp, it's better for LFS to not store Mercurial
filelog metadata, which is currently used to store rename information. That has
many advantages:

  - Large blobs could be reused across renames
  - No need to special handle files starting with `\1\n`
  - P4 LFS server implementation is much easier
  - remotefilelog LFS support is easier and cleaner

That said, the rename information is stored as lfs metadata using the
non-standard `x-hg-copy`, `x-hg-copyrev` keys. So they still exist and are
functional.

The disadvantage is that rename gets no longer hashed, which is probably fine.


Test Plan: Added a test

Reviewers: davidsp, #sourcecontrol, rmcelroy

Reviewed By: rmcelroy

Subscribers: jsgf, rmcelroy, stash, mjpieters, davidsp

Differential Revision: https://phabricator.intern.facebook.com/D4849764

Signature: t1:4849764:1491580506:1d80ad476b9cbd6773843cb52aee6745f478a0b0
2017-04-07 18:29:35 -07:00
TJ
9b91e50f77 edrecord: allow specifying a different editor for chunk selection
Summary:
a user may want to use a special-purpose 'editor' to do chunk selection, but not
have to use that same 'editor' to write commit messages and resolve merge conflicts

This is hacked together to make it work.  I'm not super familiar with the correct way to do this in python.

Test Plan: added test case

Reviewers: rmcelroy, simonfar, most, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4832207

Tasks: 15728734

Signature: t1:4832207:1491519433:82a3e62d5ae03f456abe97caf1bd352a02fe35c3
2017-04-06 16:49:27 -07:00
Jeroen Vaelen
8bc1e221b6 hgext: rename show.py to fbshow.py to prevent upstream collision
Summary:
Upstream recently introduced show as an extension. Because the file
name is the same and it lives in core, extensions.show now points to it.

It behaves differently. Rename our extension. In the next diff I'll update
facebook.rc to make extensions.show point to our version.

Test Plan: Ran tests

Reviewers: durham, #sourcecontrol

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4845959
2017-04-06 13:36:18 -07:00
Jun Wu
de84869150 lfs: cleanup user-facing messages
Summary:
The diff cleans up messages shown to the user. It makes verbose messages
gated by `if ui.verbose`, and simplifies some words.

The resulting user experience is, when there is no large file involved, lfs
shows nothing. When there are largefiles being downloaded or uploaded, show
progress bar if it takes long. The progress bar is the only user visible
output from lfs by default.

Test Plan: `rt test-lfs.t`

Reviewers: #mercurial, simonfar

Reviewed By: simonfar

Subscribers: simonfar, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4813297

Signature: t1:4813297:1491213089:c9ec363eb65df4c85282c90b230da3321b93b5e0
2017-04-05 15:58:56 -07:00
Jun Wu
f0b6b81970 lfs: simplify remote blobstore.logic
Summary:
Previously, the remote blobstore could be either `git-lfs` or `dummy`. The
application code does not really care, it only wants a "remote" blobstore.

This diff adds a factory method and makes `git-lfs` and `dummy` stores private.

The `@staticmethod get(vfs)` interface is also removed as it's duplicated and
unnecessary - as long as mercurial calls `reposetup`, the blob store objects
are set, and they cannot be missing.

The error message about an unsupported store is also changed to be consistent
with mercurial style. A test was added to test the error.

Test Plan: `rt test-lfs.t`

Reviewers: #mercurial, simonfar

Reviewed By: simonfar

Subscribers: simonfar, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4813023

Signature: t1:4813023:1491382755:b9d8ec6518141d0ba8263e16c53f430ce80c39f0
2017-04-05 15:48:10 -07:00
Stanislau Hlebik
a50af234cb infinitepush: run backup check before backup from time to time
Summary:
It's a good way to check if backup is consistent. If backup is not consistent
then we set special exit code that will be logged. It let's us easily find users
with inconsistent backups and fix it.

Test Plan: arc unit

Reviewers: #mercurial

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4810442

Tasks: 15389402
2017-04-05 00:56:28 -07:00
Stanislau Hlebik
12a13d9b3a infinitepush: do not backup extinct heads
Summary:
Previously we backed up extinct node if bookmark pointed to them. But this not
is not present in the bundle because `findcommonoutgoing()` intentionally skips
extinct and secret commits. In this case we'll have infinitepush bookmark that
points to non-existent infinitepush commit. To fix it let's filter extinct heads

Test Plan: arc unit

Reviewers: #mercurial

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4803322

Tasks: 15389402
2017-04-05 00:56:28 -07:00
Jun Wu
6d19f83af9 lfs: add a "bypass" config option
Summary:
The bypass option limits lfs's functionality to only skip hash checks. It is
intended to be used server-side, to make it more predictable - the server
never interacts with the lfs blob service.

Test Plan: Added a test case

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, simonfar, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4811828

Signature: t1:4811828:1491301114:c2e3c4200ce4cc84b9c5872a8b9a040176bb002a
2017-04-04 16:08:36 -07:00
Stanislau Hlebik
26cd83a2ee fixes: fix tests again
Summary:
fbhistedit changes was reverted upstream, revert it here too.

remove username from the test

Test Plan: arc unit

Reviewers: #sourcecontrol, quark

Reviewed By: quark

Subscribers: quark, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4830371

Signature: t1:4830371:1491343205:9d7aebd603136fb90ccd3199df60c3e0a5772428
2017-04-04 15:00:52 -07:00
Jun Wu
6c51d725bd errorredirect: log and fallback to old error handler when appropriate
Summary:
Previously errorredirect swallows errors in some cases. This diff makes it
more robust.

  - Always log commandexception. `blackbox.log` will probably catch it.
  - Print the trace if the handler script cannot be executed.
  - Print the trace if the handler script exits with non-zero.
  - Add exception for the Ctrl+C case, to not pollute user's terminal.

Test Plan: Added some test cases

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4824767

Signature: t1:4824767:1491299807:3fd863cb99ce71a6c2b59643b0c89eece7985d4b
2017-04-04 10:27:30 -07:00
Stanislau Hlebik
9e62d99f6f histedit: fix failures
Summary: Revert of D4762189 since the histedit changes were reverted upstream

Test Plan: arc unit

Reviewers: #sourcecontrol, quark, mjpieters, mitrandir

Reviewed By: mjpieters, mitrandir

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4826679

Signature: t1:4826679:1491319318:db10a0e539cdb9f0a45fe70fe93fe96daf530adb
2017-04-04 08:37:07 -07:00
Stanislau Hlebik
4cfa95492e pushrebase: fix tests
Summary:
7687edc2cef224a3a5ad4961e11073aa2ea73af8 added HG_HOOKNAME and HG_HOOKTYPE info
Let's update the test

Test Plan: arc unit

Reviewers: #sourcecontrol, mitrandir

Reviewed By: mitrandir

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4826767

Signature: t1:4826767:1491319207:a1c9e9af20cc237ff02bedcb9072f2c0816a1fbe
2017-04-04 08:36:10 -07:00
Stanislau Hlebik
ebde6a9a79 gitlookup: fix lookup if not string is passed
Test Plan:
arc unit

Note: making sure that int is passed in repo.lookup(...) is not easy.
For example `hg log -r 1` does not pass an int value to repo.lookup().
But `hg bundle` actually does it so use `hg bundle` to verify that fix works.

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4825996

Signature: t1:4825996:1491292671:9bd0897205dff93cb6dee75498827b3826f894da
2017-04-04 01:04:40 -07:00
Jun Wu
08d017f89e absorb: move to a package
Summary:
The ideal interactive mode couldn't be implemented trivially. Move `absorb`
to a directory so we can add related, but decoupled components as separate
files.

Test Plan: `make local`

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4813909

Signature: t1:4813909:1491211561:e9c40b1242c9b74230c0b8937723a2d4548e22c3
2017-04-03 10:40:31 -07:00
Simon Farnsworth
9b44edde6f fb-hgext: fix tests to account for changes to Phabricator interaction
Summary:
@simpkins fixed up some of the tech debt accumulated in our
Phabricator interfaces, which results in a minor change to output. Most tests
are fixed, but this one got missed.

Fix the test to match his fixes.

Test Plan: Run the fixed test locally

Reviewers: #sourcecontrol, stash

Reviewed By: stash

Subscribers: mjpieters, simpkins

Differential Revision: https://phabricator.intern.facebook.com/D4809914

Signature: t1:4809914:1490955858:bd2c942b9744a5529e8f4fd87bfb50666b53ae6a

Blame Revision: HGEXT8a071fbd26ed6fabf443953183c507606b027bbc
2017-03-31 03:26:04 -07:00
Jun Wu
abb7a83354 lfs: make tests stronger
Summary:
This diff makes the `test-lfs.t` much more stronger. It reveals a lot of core
hg issues in this area. I'll send patches to fix them all.

Regarding on lfs, there are some changes:

  - An existence check in its push hook was added. Otherwise pushing a revision
    with rename will cause crash.
  - The "read" processor is responsible for downloading blobs, and translate
    raw revision to lfs text. It should always return lfs text. But it may
    return raw revision text on error currently. That error handler was
    removed to avoid further damage.


Test Plan: Added new test cases. I also added `hg verify` to sanity check things are good.

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4792641

Signature: t1:4792641:1490868946:8f09c84dc9ebda3889d6a1ea04c49a06acbf38a3
2017-03-30 16:41:46 -07:00
Adam Simpkins
36dcf86559 phabstatus: fail gracefully if necessary arcrc settings are missing
Summary:
If the user does not have necesary credentials defined in their arc
configuration, catch the KeyError and convert it into an ArcConfigError.

The existing call sites in the phabstatus and arcdiff extensions catch and
handle ArcConfigError, but not generic KeyErrors.

This also fixes the phabstatus warning messages to end with a newline.

Test Plan: Added a unit test.

Reviewers: #sourcecontrol, quark, simonfar, wez, rmcelroy

Reviewed By: wez, rmcelroy

Subscribers: rmcelroy, net-systems-diffs@fb.com, yogeshwer, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4800977

Tasks: 17002914

Signature: t1:4800977:1490847078:e18bba042e3ff57100e0a7b25c610b5cad17fa2e
2017-03-30 11:55:39 -07:00
Simon Farnsworth
a8febfe71c treemanifest: add a glob to handle the argument change in the traceback
Summary:
The traceback changed due to an upstream change, but it's still doing
what we expect.

Change the test to be less sensitive to code changes.

Test Plan: `../../hg-crew/tests/run-tests.py` locally.

Reviewers: #mercurial, quark, rmcelroy

Reviewed By: rmcelroy

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4802376

Signature: t1:4802376:1490872606:8c9b357de7bd65a33abf1d5a3b0568751210aad4
2017-03-30 04:17:12 -07:00
Stanislau Hlebik
8c7c7319dd infinitepush: record nodesmetadata
Summary: Let's add a table that will store metadata about infinitepush nodes.

Test Plan: arc unit

Reviewers: #mercurial

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4778867

Tasks: 16899722
2017-03-30 02:07:30 -07:00
Stanislau Hlebik
526184c5c4 infinitepush: pass reponame to 'DELETE from bookmarkstonode...' query
Summary:
I forgot to insert it, fixing it now. It didn't cause problems before because
we delete bookmarks only during backups and backups usually have different
bookmark names

Test Plan: arc unit

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4795310

Tasks: 15389402

Signature: t1:4795310:1490810441:0a044ebe776c65b03c633017c33771d32bfa3e77
2017-03-30 01:16:06 -07:00
Stanislau Hlebik
2fc45e80f1 gitlookup: fix gitnode() template and revset
Summary:
Neither gitnode() template nor gitnode() revset worked on the server. They
both called `hg.peer(repo, {}, 'default)` and since 'default' is not a valid
url these calls failed. To fix it let's wrap `localrepo.lookup()` and call
local repo instead of peer repo if we can't find default path.

Test Plan: arc unit

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4794488

Tasks: 16848812

Signature: t1:4794488:1490795600:24269507f24613883406f0c1838fce302f86df58
2017-03-29 07:13:41 -07:00
Stanislau Hlebik
1c556e7a8c infinitepush: move sql schemas to schema.sql file
Summary: Let's avoid copy-paste.

Test Plan: arc unit

Reviewers: #mercurial, rmcelroy, quark

Reviewed By: rmcelroy, quark

Subscribers: rmcelroy, quark, mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4778949

Signature: t1:4778949:1490700154:51f6144cdb7682d03c0f6af88691fd907178319c
2017-03-29 03:21:30 -07:00
Stanislau Hlebik
e094644fb2 infinitepush: wrap findcommonincoming if nodes are requested
Summary:
Previously we don't set findcommonincoming to True, and that bug was unnoticed
because we send this info anyway in server-side `getbundlechunks()` function.

But next diff in the stack uses a fast path server-side which won't work if
fincommonincoming is not set to True.

Test Plan: arc unit

Reviewers: #mercurial, durham

Reviewed By: durham

Subscribers: durham, mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4770830

Tasks: 15389402

Signature: t1:4770830:1490719209:c251002d992e244580b7dbfeca0c30dff95734d2
2017-03-29 03:10:12 -07:00
Stanislau Hlebik
099ba152f5 infinitepush: add getavailablebackups command
Summary:
Command to list available backups for the user.
--json will be used by automation

Test Plan: arc unit

Reviewers: #mercurial, simonfar

Reviewed By: simonfar

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4728729

Tasks: 15389402

Signature: t1:4728729:1490005762:41b9683cb7dfc9d84ae1032c807d1f0c3fe60dbf
2017-03-29 03:00:46 -07:00
Adam Simpkins
675d0280a6 phabdiff: only match at the start of a line
Summary:
Only match the "Differential Revision" label at the start of a line.  We
have some diffs that include legitimate-looking Differential Revision label
strings inside parts of their test plan, which previously confused the phabdiff
output.

Test Plan: Included a unit test.

Reviewers: #sourcecontrol, quark, akushner

Reviewed By: akushner

Subscribers: net-systems-diffs@fb.com, yogeshwer, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4793483

Signature: t1:4793483:1490756514:05b2c49d5d65e1a3eadd5eb78ed0b91ef3bea42c
2017-03-28 20:53:07 -07:00
Phil Cohen
154230c8b0 pushrebase: add a test for manifests
Summary:
From diff:

```
This test does two things:

1/ Simulate a common condition of pushrebase under load. Normally pushrebase
caches data before acquiring the write lock (if lazy locking is enabled).
Under load, however, when a push has to wait for the lock more often than not,
much of this data becomes out of date and must be refetched once the lock is
acquired. This test simulates that particular case.
Specifically, we create two clients, client1 and client2, both with
nonconflicting changesets to push. client1's push is artificially blocked by a
`prepushrebase` hook (post-caching, pre-lock) that is only released after
client2's push succeeds.

2/ Checks how often we call manifest.read() inside the lock (and outside).

This way we can prevent regressions on manifest reads and test improvements.
manifest.read() is wrapped by an extension that prints a short trace. read calls
inside the lock are marked with a ":(".
```

This test is a bit of a doozy, simplification suggestions are encouraged.

Test Plan: this is, in fact, a test

Reviewers: #mercurial, durham

Reviewed By: durham

Subscribers: quark, rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4785464

Signature: t1:4785464:1490737637:79d77f6b2e49a428d63b8e1b13d769c82149ee72
2017-03-28 17:18:59 -07:00
Jun Wu
eeeb0210ae lfs: allow blobstore to be outside the repo
Summary:
Previously `lfs.blobstore` must be a relative path. If an absolute
blobstore path is set, it will traceback because vfs audit fails:

```
  File "hg/mercurial/revlog.py", line 1356, in _processflags
    text, vhash = writetransform(self, text)
  File "fb-hgext/hgext3rd/lfs/wrapper.py", line 69, in writetostore
    blobstore.local.get(self.opener).write(storeid, chunk)
  File "fb-hgext/hgext3rd/lfs/blobstore.py", line 41, in write
    fp = self._opener(self.filename(storeid), 'w+', atomictemp=True)
  File "hg/mercurial/vfs.py", line 344, in __call__
    self.audit(path)
  File "hg/mercurial/pathutil.py", line 64, in __call__
    raise error.Abort(_("path contains illegal component: %s") % path)
  Abort: path contains illegal component: /home/quark/lfslocalblobstore/d7/dbc611df1fe7dfacfe267a2bfd32ba8fc27ad16aa72af7e6c553a120b92f18
```

That was because the code was using `repo.vfs`. This diff adds a new `lfsvfs`
to avoid the issue. The `lfsvfs` also did the correct filename check (the
old `re.match` check will not match the whole string), so `blobstore.local`
could be simplified a lot.


Test Plan:
A new test case was added to make sure absolute blobstore path works. I also
did some cleanups for the test file to de-dup hgrc, and avoid writing files
outside `$TESTTMP`.

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, mjpieters, remi

Differential Revision: https://phabricator.intern.facebook.com/D4785084

Signature: t1:4785084:1490693306:b42eef8e53af107897c2e1bc2984c090bdd2e465
2017-03-28 15:41:59 -07:00
Mateusz Kwapich
3f719913eb fbmetaedit: make it compatible with mutable-history without our hotfixes
Summary:
It's the upstream mutable-history that handles the "--fold" case now.
It doesn't have the fix that avoids touching wd in that case.

Test Plan:
tests are passing when mutable-history is updated to default-upstream-tip
I'll delete the actual mutable-history hotfixes once this is accepted.

Reviewers: #sourcecontrol

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4779366
2017-03-27 08:08:27 -07:00
Jun Wu
c3fd525bf8 lfs: fix tests
Summary:
A bunch of modifications to get the test pass with the new lfs code.

- Move `lfs` to `hgext3rd`. The code was supposed for hg-core. For now, we do
  them in fb-hgext to speed up the process
- Remove the windows test, which is not supported by `run-tests.py` and is
  duplicated with `test-lfs.t`.
- Do import `mercurial.i18n._` correctly.
- Change some i18n logic a bit so it's more translator-friendly.
- Change `revlog.RevlogError` to `error.RevlogError`.
- Avoid direct symbol import of `mercurial.util.bytecount`, which will fail the
  upstream importchecker test.
- Fix various lint issues like lines being too long etc.
- Document lfs config options.

Test Plan: `arc unit`

Reviewers: #sourcecontrol, rmcelroy

Reviewed By: rmcelroy

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4772216

Signature: t1:4772216:1490401458:1ad3c18ab80e1d31085d0b6b4c630e62a7dc7930
2017-03-24 19:01:42 -07:00
Remi Chaintron
dcdf35ff63 lfs: new extension
Add the lfs extension and its tests.

----
quark: This imports @remi's lfs work for core hg. See D3980880, D4770352,
and D4771511.
2017-03-24 19:00:30 -07:00
Simon Farnsworth
ccf6655f3a fb-hgext: fix test-check-code.t failures
Summary: test-check-code.t now dislikes our docstrings. Fix them up

Test Plan: Run the test locally on my devserver

Reviewers: #sourcecontrol, durham

Reviewed By: durham

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4773908

Signature: t1:4773908:1490394131:9d9e5f85b4243119a3615044605bcc5abdecbe4f
2017-03-24 15:22:31 -07:00
Stanislau Hlebik
b628d6cc32 sqldirstate: fix sqldirstate-fbhistedit test
Summary: Just as in D4762189. Fixes to match upstream changes

Test Plan: arc unit

Reviewers: #sourcecontrol, simonfar

Reviewed By: simonfar

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4762596

Signature: t1:4762596:1490286227:004ed0a7055aab64769893e80a995975b62d5ea0
2017-03-23 09:25:45 -07:00
Simon Farnsworth
0727779c39 fbhistedit: fix tests after core change
Summary:
changeset:   06d6fecfdb713e3a2e2013de046af0bce4005b96   (@)
user:        Jun Wu <quark@fb.com>
date:        Mon, 13 Mar 2017 22:19:06 -0700

    histedit: use safecleanupnode in _finishhistedit

        This simplifies code a lot.

causes significant changes to the test output. Update to match

Test Plan: Run tests locally

Reviewers: #sourcecontrol, quark, stash

Reviewed By: stash

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4762189

Signature: t1:4762189:1490281032:6a48429f60584c4318bd1e247ce2eeb2775d623c
2017-03-23 07:57:35 -07:00
Durham Goode
f108f54ab0 fixtests: fix morestatus and perftweaks tests to match upstream
Upstream changed the transaction logic around rebase, and added better quoting
to tests, so we need to update our internal tests accordingly.
2017-03-21 09:18:01 -07:00
Durham Goode
2393924384 fastmanifest: add __bool__ to hybridmanifest
Upstream added __bool__ to manifestdict, and the test checking that
hybridmanifest matches the functions of manifestdict started failing. Let's add
it.

The check-code.py change matches the one in upstream to support the addition of
__bool__.
2017-03-20 18:58:10 -07:00
Durham Goode
0942e94dcc treemanifest: reverse historypack entries
History pack entries need to be in newest-to-oldest order, so let's reverse
them.
2017-03-20 18:58:09 -07:00
Jun Wu
1e30f781e0 dirsync: rename in-repo config file to .hgdirsync
Summary:
It will allow sparse to not ignore the file, and is more consistent with
other special files.

Test Plan: Updated existing tests

Reviewers: rmcelroy, #sourcecontrol, durham

Reviewed By: durham

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4738766

Signature: t1:4738766:1490032493:199f3fef9c74a137b16ae7637b87de625ca5115d
2017-03-20 10:55:43 -07:00
Stanislau Hlebik
f0b136f93b newhiddencache: add knob to enable fasthiddencache
Summary: Add a knob to easily disable/enable fasthiddencache functionality

Test Plan: arc unit

Reviewers: #mercurial, mjpieters

Reviewed By: mjpieters

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4714509

Tasks: 16558840

Signature: t1:4714509:1489717542:d022201fd7c17863ffe664ae4115b8d1d896da3c
2017-03-20 01:38:06 -07:00
Stanislau Hlebik
32199b95b4 newhiddencache: restore back
Summary:
This diff just restores the code that was reverted before. Next diffs will fix
it

Test Plan: arc unit

Reviewers: #mercurial, mjpieters

Reviewed By: mjpieters

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4714397

Tasks: 16558840

Signature: t1:4714397:1489717436:9777d783d1443dea46460450e8f3da862c147483
2017-03-20 01:38:06 -07:00
Durham Goode
63f4fa69a0 datapack: remove delta reuse
Summary:
This code that reused deltas if the delta parent wasn't available was bugged
because it meant you could end up with a cycle in the delta chains. This was an
old optimization from before trees had history, so let's drop the optimization
(since trees now have history and can be correctly repacked).

Test Plan:
Ran repack on a packfile that previously caused cycles. Verified the
new version did not with `hg debugdatapack foo.datapack'

Reviewers: #mercurial

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4724520
2017-03-18 19:38:45 -07:00
Jun Wu
5404cd9a3c codemod: get rid of ui.backupconfig
Summary:
The upstream will remove `ui.backupconfig` soon(tm). Let's migrate our code
to `ui.configoverride` to avoid future breakages.

Test Plan: `arc unit`

Reviewers: #sourcecontrol, simonfar

Reviewed By: simonfar

Subscribers: simonfar, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4734300

Signature: t1:4734300:1489803960:a4101d6bc95ab62af67f0076b8e21f57cf926e13
2017-03-17 19:42:50 -07:00