Commit Graph

2623 Commits

Author SHA1 Message Date
Durham Goode
f2405cda64 treemanifest: update finalize tests
Summary:
The current tests test some undefined behavior of manifest.finalize() where it
can rewrite immutable Manifest instance's with a new node. This happens if you
call finalize without passing the appropriate parent that the manifest was
copied from. So the new tree contains unmodified subtrees from the original
manifest, and since we don't pass the original manifest to finalize, it doesn't
realize these trees already existed and tries to give them a new node.

A future patch makes tree mutability more strict and caught this. Fixing the
tests now, so I can show that future patches don't break this.

Test Plan: Ran the test

Reviewers: #mercurial, stash

Reviewed By: stash

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4880057

Signature: t1:4880057:1492589057:a7026bc791ea2cd12aa790513cf46d5180cb0638
2017-04-19 21:14:03 -07:00
Durham Goode
c1f233a304 treemanifest: rename NewTreeIterator to SubtreeIterator
Summary:
The NewTreeIterator class was used for iterating over a new tree, when given
it's p1. Mainly for performing the write of a new tree. We want to make this a
more generic mechanism so we can do the diff of one tree with an arbitrary
nother (like when deciding what trees to send a client if they have a given base
node). Let's start by renaming the class.

Test Plan: Ran the tests

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4880048

Signature: t1:4880048:1492110564:bd51d35e4cfcca2d6b1d4cf81d78fdacf2b52f3d
2017-04-19 21:14:03 -07:00
Durham Goode
8da9d7b37f treemanifest: automatically download missing trees on demand
Summary:
This adds a remote tree datastore to the store, so if we don't have a tree on
local disk, we can reach out to the server for it.

This also makes a small change to fastmanifest to use get() instead of
getmissing() to test if the tree is available. getmissing() would indicate the
tree is missing if it wasn't local, while get() will attempt to fetch it from
the server.

Test Plan: Added a test

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: quark, stash, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4864645

Signature: t1:4864645:1492627823:7afff5db54c895507507e57ab7263f4dbf10b5f7
2017-04-19 21:14:03 -07:00
Durham Goode
b44263e197 fixtests: move treemanifest configs onto client only
Summary:
The treemanifest tests were setting the configs on both the client and the
server, without telling the server it was a server and without setting it up
appropriately to be a client. This caused the tests to fail in a future patch.
Let's move the treemanifest config to the clients, since that's the only bits
being tested here.

Test Plan: Ran the tests

Reviewers: #mercurial, ikostia

Reviewed By: ikostia

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4864766

Signature: t1:4864766:1491903651:ab91eed5a2eb8a60db8e230c2ba5c0ee29b5adc9
2017-04-19 21:14:03 -07:00
Durham Goode
3633627260 treemanifest: add prefetchtrees for receiving trees from the server
Summary:
This adds a new prefetchtrees command and gettrees wireprotocol command for
receiving trees from the server. The actual wire protocol is a custom request
format, with a normal pack file response format (same as used in remotefilelog).
The request args can specify which manifest nodes are wanted, for which
directory and which subdirectories.

I attempted to use the changegroup3 response format instead of the pack format,
but changegroup is extremely intertwined with revlogs and with producing
changegroups for linear swaths of the repo history, instead of random parts of
just the manifest.

Test Plan: Added a test

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: stash, quark, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4864435

Signature: t1:4864435:1492626392:c3603eeb261a1f9b9d836f0d44614025b58d45ea
2017-04-19 21:14:03 -07:00
Arun Kulshreshtha
02e5cc57ba tweakdefaults: allow hg update --merge without --nocheck
Summary: The --merge flag is relatively new; we shouldn't add --check if it is specified.

Test Plan: Unit test

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4916396

Tasks: 17490374

Signature: t1:4916396:1492647440:d9955931cb9177719a58325c08a715d4d51d1226
2017-04-19 17:20:23 -07:00
Arun Kulshreshtha
6065ceef3f fbamend: use unfiltered repo when clearing preamend bookmarks
Summary: Sometimes restack attempts to clear preamend bookmarks on hidden commits. This only happens when the user has a configuration error (typically inhibit is disabled) which would mess up other parts of restack. Nonetheless, there is no reason to crash here.

Test Plan: Disable inhibit, perform an amend, and run `hg restack`. Unit test is forthcoming.

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4856640

Tasks: 16726367

Signature: t1:4856640:1491816803:56beeb11360a7a2f6bc6243069c562b08a015afc
2017-04-19 15:52:45 -07:00
Wez Furlong
04ee6433d4 warnings: squelch contextlib.nested DeprecationWarnings
Summary:
deals with the tests failing like this on my python 2.7 system:

```
+  /data/users/wez/facebook-hg-rpms/fb-hgext/hgext3rd/uncommit.py:144: DeprecationWarning: With-statements now directly support multiple context managers
+    with nested(repo.wlock(), repo.lock()):
```

Test Plan: arc unit

Reviewers: #mercurial, stash, simonfar, wez

Reviewed By: stash, simonfar

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4910040

Signature: t1:4910040:1492593520:ff815c5dbed4a341e0a313cd7160b700d878ee2c
2017-04-19 03:28:23 -07:00
Simon Farnsworth
e31246ce99 rage: include klist output
Summary:
We're seeing a rise in the number of auth issues. As a first step,
let's have rage grab klist, so we can see if the user has a valid Kerberos
ticket for hg.vvv or not.

Test Plan: Run it and see https://phabricator.intern.facebook.com/P57315650

Reviewers: #sourcecontrol, ikostia

Reviewed By: ikostia

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4907615

Signature: t1:4907615:1492546856:969cab7b8deacf220c374d4c9e1b6e99f84ca059
2017-04-19 03:18:04 -07:00
Kostia Balytskyi
aeef0ad5cc compatibility: migrate from scmutil.vfs to mercurial.vfs.vfs
Differential Revision: https://phabricator.intern.facebook.com/D4908906
2017-04-18 14:42:33 -07:00
Wez Furlong
934dfaf08b templater: fixup for templater changes upstream
Summary:
The function signature for a number of functions relating to templating changed.

In addition, the templater layer now requires that template functions have been
registered via the decorator, otherwise a require attribute is missing and
things blow up.

Test Plan: arc unit.  rt.

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4908220

Signature: t1:4908220:1492545033:66f089e03a6064ae96ca36f3abf99ecb6d48d422
2017-04-18 13:04:11 -07:00
Wez Furlong
7170cd6411 tweakdefaults: switch hg grep to use dirstate.walk
Summary:
previously, this was using .match() to find candidate names
and then reaching inside the dirstate._map to check the status of
the file.

This implementation uses the dirstate.walk instead, which returns a
map of filename -> stat information.

This approach works better with custom dirstate implementations, such
as eden, where a local map of the required information is not available
and where it would be suboptimal to request that information on
demand on a per-file basis.

Test Plan:
testing before and after this change:

`time hg grep eden foo` has a runtime of ~1.2 seconds.

If you have sqldirstate enabled, this change makes the runtime 3 seconds
compared to 2 with sqldirstate on and without this change.

Reviewers: #mercurial, durham

Reviewed By: durham

Subscribers: mjpieters, simpkins

Differential Revision: https://phabricator.intern.facebook.com/D4903110

Signature: t1:4903110:1492473495:ebe375f3e2533f6ae4ca7ea9a494f6feb9704f9e
2017-04-18 10:05:31 -07:00
Stanislau Hlebik
928a96b602 infinitepush: use ui.debug() instead of ui.warn()
Summary: To avoid spamming users

Test Plan: Run tests

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4883447

Signature: t1:4883447:1492088221:3649b14ff514957d2f412aae2647ca1b20ef5fea
2017-04-17 23:44:26 -07:00
David Soria Parra
61c9ea8ec7 p4fastimport: require p4 during tests
Summary: Require perforce during all tests.

Test Plan:
run tests with and without Perforce in PATH. Tests correctly run with
P4D and P4 in path and were correctly skipped without.

Reviewers: #sourcecontrol, #idi, durham

Reviewed By: durham

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4899085

Signature: t1:4899085:1492475858:3bd1443c707e56461835d278a9c6bf3e034b5f4a
2017-04-17 18:08:07 -07:00
Arun Kulshreshtha
a72a47d515 absorb: s/chunks/chunk
Summary: Noticed this minor gramatical error when using absorb.

Test Plan: Unit tests.

Reviewers: quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4887040

Signature: t1:4887040:1492123047:dcae53cabd64f1df13d80a20953c712ce0aaaa09
2017-04-13 16:00:51 -07:00
David Soria Parra
ef08c10f5b p4fastimport : introducing fast Perforce to Mercurial convert extension
Summary:
`p4fastimport` is a fast convert extensions for Perforce to Mercurial. It
is designed to generate filelogs in parallel from Perforce. It tries to
minimize the use of Perforce commands and reads from the the Perforce
store on a Perforce server directly.

The core of p4fastimport is the idea to generate a Mercurial filelog
directly from the underlying Perforce data, as a Perforce file in most
cases matches a filelog directly (per-file branches is an exception). To
generate a filelog we are reading each file for an imported revision. A
file in Perforce is locally either stored in RCS, as a compressed GZIP
or as an flat file (binaries). If we do not find a version locally on
disk we fallback to downloading it from Perforce.

We are generating manifests after all filelogs are imported. A manifest
is constructed by adding and removing files from an initial state. We
are generating the correct offset from a manifest into the filelog by
keeping track of how often a file was touched.

We then generate the changelog.

Linkrev generation is a bit tricky. For every file in Perforce know
to which changelist it belongs, as it's stored revisions contains the
changelist. E.g.  1.1422 is the file changed in the changelist 1422 (this
refers to the "original" changelist, before a potential renumbering,
which is why we use the -O switch).  We use the CL number obtained
from the revision to reverse lookup the offset in the sorted list of
changelists, which corresponds to it's place in the changelog later,
and therefore it's correct linkrev.

Parallel imports: In order to run parallel imports we MUST keep one lock
at a time, even if we import multiple file logs at the same time. However
filelogs use a singular `fncache`, which will be corrupted if we generate
filelogs in parallel. To avoid this, repositories must be generated with
*fncache* disabled! This restricts `p4fastimport` with workers to run
only on case sensitive file systems.

Test Plan:
The included tests as well as multiple imports from a small testing
Perforce client. Afterwards successfully run `hg verify`

  make tests

Reviewers: #idi, quark, durham

Reviewed By: durham

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4776651

Signature: t1:4776651:1492015012:0161c4f45eab4d3b64597d012188c5f2007e8f7d
2017-04-13 11:11:09 -07:00
Stanislau Hlebik
57dc185f42 infinitepush: don't make separate backups for different working copies
Summary:
Previously we had separate backup per working copy. That's very confusing
since all these working copies shares the same repo. This diff fixes it and
also adds config option to clean unnecessary working copy server-side.

Test Plan: Run infinitepush unittest

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: rmcelroy, quark, mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4876230

Signature: t1:4876230:1492025747:3579e5046efc2ed309044fc3335c36ac4f7bdd04
2017-04-13 04:58:11 -07:00
Wez Furlong
4a59f3b701 c-extensions: fixup some compiler/environment portability concerns
Summary:
I sync'd a copy of this code into the eden repository.
I had to adjust a couple of include paths to get the code to
compile correctly in the hermetic build environment that is
in use there.

In addition, our linter suite over there found a couple of C++ nits
to be fixed up.

Test Plan: make local

Reviewers: simpkins, ikostia, simonfar, durham

Reviewed By: durham

Subscribers: net-systems-diffs@fb.com, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4879285

Signature: t1:4879285:1492039044:8cb1e033e35ee568806de94dda3d2f6f8e78f5cb
2017-04-12 16:34:53 -07:00
Adam Simpkins
642dea7e2c fixcorrupt: automatically look back as far as necessary
Summary:
Update the fixcorrupt extension to look back as far as necessary in the revlog
to find a good entry.  Previously the user had to supply a good value using the
--checklen argument.  If --checklen was too small, fixcorrupt could end up
trying to truncate in the middle of a corrupted section of the revlog, which
would cause problems as it was still using corrupted data.

This change ensures that fixcorrupt always looks back far enough to find a good
entry.  We start by looking at the last 10 entries, and double that amount each
time the first item in the list is still bad.

Test Plan:
Used it to fix a corrupted fbsource repository where the corruption was 36
commits back.

Reviewers: #mercurial, quark, stash

Reviewed By: stash

Subscribers: stash, net-systems-diffs@fb.com, yogeshwer, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4874243

Signature: t1:4874243:1492007518:7406b03a9967815a496a8c2fae394c5f21f7e60a
2017-04-12 14:09:22 -07:00
Durham Goode
4da0040f03 fastlog: fix ScmQuery request parameter name
Summary:
This was totally wrong and has been forever. I think we got lucky because the
server was serving from memcache mostly, which did not validate this parameter.

Test Plan:
Ran log before and after the change.

hg.real log wdc --config extensions.fastlog=../../facebook-hg-rpms/fb-hgext/hgext3rd/fastlog.py

Reviewers: #mercurial, mitrandir

Reviewed By: mitrandir

Subscribers: mjpieters, jeroenv

Differential Revision: https://phabricator.intern.facebook.com/D4878633

Tasks: 17327887

Signature: t1:4878633:1492030224:cf1b802345c7e00d1134ee81fbe9f271cf1a6752
2017-04-12 13:51:18 -07:00
Mateusz Kwapich
75778a9b81 remotefilelog: fix tests
Summary: Only one exact form on "hg" invocation with --stdio is now allowed

Test Plan: tests are passing now

Reviewers: #mercurial, simonfar

Reviewed By: simonfar

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4877162
2017-04-12 13:21:16 -07:00
Stanislau Hlebik
8f8cdc2d49 hgext3rd: cleanobsstore extension
Summary: See comments and docstring

Test Plan: arc unit

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: quark, mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4851138

Signature: t1:4851138:1491589247:5ca750e45bcc9fc996c9d7c14e31cb5f24ea983f
2017-04-12 01:16:01 -07:00
Stanislau Hlebik
efbec7652d infinitepush: fix security concern
Summary: Check owner of a log file before writing to it. See comments for details

Test Plan: arc unit

Reviewers: #mercurial, simpkins

Reviewed By: simpkins

Subscribers: net-systems-diffs@fb.com, mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4842933

Tasks: 17155924

Signature: t1:4842933:1491861137:e8a027fb7a930c0c5f553c75cb84214d24f66ce3
2017-04-12 01:12:23 -07:00
Jun Wu
e0eb41504b test-fixcorrupt: be OSX compatible
Summary:
`wc` in OSX has different padding behavior that breaks the test:

```
       $ wc -c .hg/store/00changelog* .hg/store/00manifest* | sort
    -   99 .hg/store/00manifest.d
    -  110 .hg/store/00changelog.d
    -  128 .hg/store/00changelog.i
    -  128 .hg/store/00manifest.i
    -  465 total
    +        99 .hg/store/00manifest.d
    +       110 .hg/store/00changelog.d
    +       128 .hg/store/00changelog.i
    +       128 .hg/store/00manifest.i
    +       465 total
```

This patch fixes it.

Test Plan: The fix was verified manually on an OS X machine.

Reviewers: #mercurial, durham

Reviewed By: durham

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4872689

Signature: t1:4872689:1491954172:92178178deea88ec3545c42ad0cdecc8099a1db6
2017-04-11 16:43:32 -07:00
Jun Wu
340750c4f4 pushrebase: send obsmarkers to client if client supports it
Summary:
Previously when the client has obsstore enabled, and the server has obsstore
disabled, clients won't get the obsmarkers, which is suboptimal. This diff
makes the server send obsmarkers in that case.

Practically, this means people will no longer need to run `strip` after landing
a diff in this repo.

Note: `test-pushrebase-manifests.t` is somehow flaky about the empty lines. I
guess they are not related to the change. But the test change is not 100%
reproducible.

Test Plan: Added a new test case

Reviewers: #mercurial, stash

Reviewed By: stash

Subscribers: rmcelroy, ikostia, stash, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4865150

Signature: t1:4865150:1491895844:afbc3079a40a9a9fa9af1eab4eeaca91091e8d2d
2017-04-11 13:21:27 -07:00
Jun Wu
11cd7af681 lfs: add a fast path for filelog.size
Summary:
Previously, `filelog.size` requires loading the lfs blob in memory. This
makes it unnecessary.

Test Plan: Run existing tests.

Reviewers: #mercurial, mitrandir

Reviewed By: mitrandir

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4861358

Signature: t1:4861358:1491923349:f1d2fe5656158854e1dd50987ea3db64a9793db6
2017-04-11 13:20:58 -07:00
Jun Wu
de5e7c6545 fixcorrupt: new extension to fix corrupted repo
Summary:
We sometimes get reports about corrupted repos. Usually the corrupted part
is just at the end of changelog or manifest.

Truncating them manually works but people need to be very careful. This
extension is like the manual fix but automatized.

Test Plan:
Run `hg debugfixcorrupt --no-dryrun` on a reported corrupted repo and check
it truncates files correctly and the repo passes `hg verify` afterwards.

Reviewers: #mercurial, durham, stash

Reviewed By: stash

Subscribers: stash, rmcelroy, durham, lcharignon, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D3408396

Signature: t1:3408396:1491897232:fc17a105124b568963441adfec97e26735df3258
2017-04-11 13:19:42 -07:00
Adam Simpkins
fdc2c3e6f9 extutil: add unit tests for runbgcommand
Summary:
Add some basic unit tests for extutil.runbgcommand().

This also changes the behavior to throw an OSError if we fail to execute the
process, instead of a subprocess.CalledProcessError.  This matches the behavior
of the subprocess module when it fails to execute the command.

Test Plan: Ran the tests.

Reviewers: stash, quark

Reviewed By: quark

Subscribers: net-systems-diffs@fb.com, yogeshwer, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4864999

Signature: t1:4864999:1491874918:c03edafe02af217e41c28a770137bfd72bcbba9b
2017-04-11 11:25:40 -07:00
Adam Simpkins
05c70db64f build: make sure portability/ is always added to the include path
Summary:
The portability/ subdirectory always needs to be listed in the compiler include
paths in order to successfully build.  This fixes setup.py to always add it to
the list of include directories, rather than only adding it when INCLUDE_DIRS
was not specified through the environment.

Test Plan:
Successfully built the fb-hgext repository when specifying alternate system
header paths through the INCLUDE_DIRS environment variable.

Reviewers: tja, wez, ikostia

Reviewed By: ikostia

Subscribers: net-systems-diffs@fb.com, yogeshwer, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4865296

Signature: t1:4865296:1491903269:fc07edf01a89117e25805e9ee936b2e2eee22410
2017-04-11 11:17:05 -07:00
Ryan McElroy
8267123b73 scm-prompt: add eden snapshots support
Summary:
Based on the recent commit wez made, but centralized to the scm-prompt
source-of-truth.

Test Plan: updated test

Reviewers: #sourcecontrol, wez, simpkins, simonfar

Reviewed By: simonfar

Subscribers: net-systems-diffs@fb.com, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4867662

Tasks: 17110799

Signature: t1:4867662:1491916786:647edb819a059a3a6baa09c8d189ea706933687b
2017-04-11 08:47:12 -07:00
Pallav Shinghal
3ded80a5c7 rage: fix hg sparse entry in hg rage output
Summary: The `hg sparse` entry in `hg rage` output is consistently `(Failed. See footnote [1])` due to the `sparse` function being called with the wrong signature.

Test Plan:
Ran `hg rage` in <reponame>. Output contained:

```
hg sparse:
---------------------------
%include .hgsparse-<reponame>
[include]

[exclude]

```

Reviewers: #sourcecontrol, quark

Reviewed By: quark

Subscribers: quark, mjpieters, dawidp

Differential Revision: https://phabricator.intern.facebook.com/D4861832

Tasks: 17269067

Signature: t1:4861832:1491869628:0abc55eb5308a83effe78943dba8d13e9f12d616
2017-04-11 07:19:26 -07:00
Durham Goode
902af15d1e remotefilelog: refactor pack wireprotocol to separate file
Summary:
The pack wireprotocol will be useful for exchanging treemanifests, so let's
refactor it out to it's own file. There's a slight protocol change here, where
we terminate the response with 10 null bytes (2 for 0 length filename, 4 for 0
length data, 4 for 0 length history) instead of the original 2 null bytes (for
0 length filename) which didn't let us handle entries with '' as the name.

This pack exchange code isn't even used in production, since most remotefilelog
downloads are done via the lose file (getfile/getfiles) format.

Test Plan:
Ran the tests. Even though this code isn't used in production, the
prefetch and repack tests still cover it.

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4860556

Signature: t1:4860556:1491853521:c3810a4a681606571354b270b957e8df0962c86a
2017-04-10 17:56:01 -07:00
Jun Wu
dc051c4791 tweakdefaults: fix common incorrect PAGER config automatically
Summary:
People may mistakenly set PAGER=less, which will lose colors and does not
have the friendly "exit if there is less than one screen" feature.

Detect that and add "-FRQX" automatically.

Test Plan:
Unset all my `pager.pager` configs, and export `PAGER=less`, verify `hg sl` has
the desired pager and colors are fine.

Reviewers: wez, #mercurial, durham

Reviewed By: durham

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4864822

Signature: t1:4864822:1491870110:246f906d41b5ac72ecfdda487aa8f412a90cf2cb
2017-04-10 17:38:22 -07:00
Adam Simpkins
df34a531e6 extutil: add a new runbgcommand() function
Summary:
Add a new `runbgcommand()` function, which is similar to `runshellcommand()`
but does not require running the command through a subshell.  Running commands
through a shell is a very common place for security vulnerabilities if care is
not taken to correctly escape all arguments.

This changes `runshellcommand()` to be implemented using `runbgcommand()`.
This does change the behavior of `runshellcommand()` slightly:
`runshellcommand()` now throws an exception if it failed to execute the
command.  Previously the Unix implementation silently swallowed most
exceptions.  This also fixes the unix implementation to properly wait on the
intermediate child process.  Previously it would remain as a zombie process.

Test Plan: Ran all unit tests.

Reviewers: #mercurial, stash

Reviewed By: stash

Subscribers: quark, net-systems-diffs@fb.com, yogeshwer, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4856672

Signature: t1:4856672:1491813879:367e1f42b9aa7797542f0d02da99f46683b80df8
2017-04-10 11:44:32 -07:00
Adam Simpkins
6d24f86d76 arcanist: support arc unit --all
Summary: Update the MercurialTestEngine and unit.py script to support running all tests.

Test Plan: Ran `arc unit --all`

Reviewers: #sourcecontrol, quark

Reviewed By: quark

Subscribers: net-systems-diffs@fb.com, yogeshwer, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4856665

Signature: t1:4856665:1491623577:da0b639121d9df6b933d160d4776ec3515a91187
2017-04-10 11:44:32 -07:00
Jun Wu
2f1756608e lfs: remove manifest walk in prepush hook
Summary:
`ctx.files` should contain changed files from both p1 and p2. There is no
need to walk the manifest, which could also be painfully slow for large
repos. So the manifest walk got removed.

Test Plan: `arc unit`

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4856552

Signature: t1:4856552:1491639067:8080f9bc7e246a8d36fcc3504716e9c95ee24fcf
2017-04-10 11:22:54 -07:00
Jun Wu
46d44c8b07 lfs: use non-chunking spec format if possible
Summary:
The "chunking" feature is not specified by [the current Git LFS standard](21e1695220/docs/spec.md).

Therefore avoid using it if possible - if there is only one chunk, use the
standard specification (`https://git-lfs.github.com/spec/v1`).

An upload message is slightly changed to be more accurate.

Test Plan: Changed existing tests

Reviewers: davidsp, #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4856527

Signature: t1:4856527:1491638822:1c6f555b706e7bb22dd9090afa156f2161bf9f7f
2017-04-10 11:17:48 -07:00
Jun Wu
38632025a9 lfs: add bundle support
Summary:
This diff adds bundle support for lfs:

  - Let `hg bundle` use changegroup3 instead of changegroup2 to record revlog
    flags.
  - Hook related functions so `hg -R bundle.hg` works with LFS.

Test Plan: Added a test

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4856335

Signature: t1:4856335:1491638660:d56ba54ea5f59262f009418b9c7e48c8f2a25ed6
2017-04-10 11:11:37 -07:00
Jun Wu
cdc9accca5 lfs: do not store mercurial filelog metadata in lfs blobs
Summary:
Per discussion with @davidsp, it's better for LFS to not store Mercurial
filelog metadata, which is currently used to store rename information. That has
many advantages:

  - Large blobs could be reused across renames
  - No need to special handle files starting with `\1\n`
  - P4 LFS server implementation is much easier
  - remotefilelog LFS support is easier and cleaner

That said, the rename information is stored as lfs metadata using the
non-standard `x-hg-copy`, `x-hg-copyrev` keys. So they still exist and are
functional.

The disadvantage is that rename gets no longer hashed, which is probably fine.


Test Plan: Added a test

Reviewers: davidsp, #sourcecontrol, rmcelroy

Reviewed By: rmcelroy

Subscribers: jsgf, rmcelroy, stash, mjpieters, davidsp

Differential Revision: https://phabricator.intern.facebook.com/D4849764

Signature: t1:4849764:1491580506:1d80ad476b9cbd6773843cb52aee6745f478a0b0
2017-04-07 18:29:35 -07:00
Stanislau Hlebik
1c6b1b495d fastpartialmatch: catch write exceptions
Summary:
We saw a couple of weird exceptions during clone. Since read path is wrapped
in try/catch, let's wrap write path too. If exception happened then let's
remove index completely since it will be rebuilt on the next pull.

Test Plan:
arc unit

Unit-tests are actually almost useless in this case.
So I added fake exception raises and checked that it behaves as expected.

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4843497

Tasks: 17153898

Signature: t1:4843497:1491536062:8e72a4ad4fffe5625866a90b3b89acda75211ce8
2017-04-07 02:21:54 -07:00
Stanislau Hlebik
dd5c816e86 fix tests
Summary:
In 46991460544739fd0ac84d496986e7d566bb8a9f _jsonifyobj method was deleted.
Now templatefilters.json(..., paranoid=False) should be used

Test Plan:
Run tests
2017-04-07 01:03:26 -07:00
TJ
9b91e50f77 edrecord: allow specifying a different editor for chunk selection
Summary:
a user may want to use a special-purpose 'editor' to do chunk selection, but not
have to use that same 'editor' to write commit messages and resolve merge conflicts

This is hacked together to make it work.  I'm not super familiar with the correct way to do this in python.

Test Plan: added test case

Reviewers: rmcelroy, simonfar, most, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4832207

Tasks: 15728734

Signature: t1:4832207:1491519433:82a3e62d5ae03f456abe97caf1bd352a02fe35c3
2017-04-06 16:49:27 -07:00
Jeroen Vaelen
8bc1e221b6 hgext: rename show.py to fbshow.py to prevent upstream collision
Summary:
Upstream recently introduced show as an extension. Because the file
name is the same and it lives in core, extensions.show now points to it.

It behaves differently. Rename our extension. In the next diff I'll update
facebook.rc to make extensions.show point to our version.

Test Plan: Ran tests

Reviewers: durham, #sourcecontrol

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4845959
2017-04-06 13:36:18 -07:00
Kostia Balytskyi
918e1011b3 portability: use COMPOUND_LITERAL macro in place of actual compound literals
Summary:
This is a s//g replacement of all the `return (type_name) {` with
`return COMPOUND_LITERAL(type_name) {`.
This is the command which produced the diff:
`egrep 'return \(\w*\) \{' -Ir . --exclude='*.py*' --exclude-dir=.hg -l | xargs sed 's/return (\(\w*\)) {/return COMPOUND_LITERAL(\1) {/g' -i `

After I've done this, I checked:
`egrep '\(\w+\) \{' -Ir . --exclude='*.py*' --exclude-dir=.hg | egrep -v '(switch|while)' | grep -v 'if (' | grep -v 'COMPOUND_LITERAL' | less`
and it looks like the only things of `(something) {` syntax are function definitions, adding space before `(` in search pattern yields no results.

This is needed to make this compile on Windows under MSVC2015.

Depends on: D4843230

Test Plan:
- run `python setup.py build -f`, see it compile
- run all the tests, see them pass

Reviewers: #sourcecontrol, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4843240

Signature: t1:4843240:1491495690:a097bfab403805052d5ac25d1db7bb32af3bca28
2017-04-06 09:34:40 -07:00
Kostia Balytskyi
67b7f56ddb portability: add a portability header
Summary:
Proposed header (or its dir) is a single place to put MSVC/GCC hacks. So
far it only includes the COMPOUND_LITERAL macro which behaves differently
depending on MSVC mode.
When MSVC2015 is used in C++ mode, it does not support things like:
`(my_type) {initializers}`, but in C mode it does.

To clarify: I am not even sure whether we need to have the ability to compile in a purely C mode, but I did not want to figure out.

Test Plan: - on Linux, run `python setup.py build`, run all the tests, see them passing

Reviewers: #sourcecontrol, tja

Reviewed By: tja

Subscribers: tja, jsgf, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4843230

Signature: t1:4843230:1491496062:3fa10ae5a5aac850689991de1ca6ee1ac86d9dce
2017-04-06 09:33:34 -07:00
Kostia Balytskyi
18c695ce6a portability: replace variably-sized C arrays with explicit mallocs
Summary:
This is an RFC kind of diff, so I am looking for comments. I am not very
familiar with either C or C++.
The goal is to make all this stuff compile on Windows under MSVC2015. MSVC
does not support variable-sized arrays in C++ out-of-the-box (not sure
whether there are tricks to make it do so), so my proposal is to use
explicit `malloc`s in place.

Test Plan:
Only tested on Linux for now. `python setup.py build` works,
tests pass, seems to make sense.

Reviewers: #sourcecontrol, jsgf

Reviewed By: jsgf

Subscribers: jsgf, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4839968

Signature: t1:4839968:1491492552:c80fac6990aaee78e6bb18522ff13e02eb9521e4
2017-04-06 09:30:26 -07:00
Stanislau Hlebik
9878a16264 infinitepush: do not rebundle if not necessary
Summary:
Profiles show that rebundling is the slowest part in infinitepush that can
take up to 10 secs to run. In many cases rebundling is not needed at all.
For example, if bundle contains only one head and that head was requested
during pull then rebundling can be avoided at all.

Test Plan: arc unit

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: quark, mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4834982

Tasks: 15389402

Signature: t1:4834982:1491450862:fe6984e6384676bcbab1c3fb9f0a47cb93902d0d
2017-04-06 02:53:28 -07:00
Jun Wu
de84869150 lfs: cleanup user-facing messages
Summary:
The diff cleans up messages shown to the user. It makes verbose messages
gated by `if ui.verbose`, and simplifies some words.

The resulting user experience is, when there is no large file involved, lfs
shows nothing. When there are largefiles being downloaded or uploaded, show
progress bar if it takes long. The progress bar is the only user visible
output from lfs by default.

Test Plan: `rt test-lfs.t`

Reviewers: #mercurial, simonfar

Reviewed By: simonfar

Subscribers: simonfar, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4813297

Signature: t1:4813297:1491213089:c9ec363eb65df4c85282c90b230da3321b93b5e0
2017-04-05 15:58:56 -07:00
Jun Wu
b1cdf3f0e6 lfs: remove setup.py
Summary:
The logic in `setup.py` is now simple enough to be moved to `reposetup`.
`setup.py` becomes unused and gets removed.

Test Plan: `rt test-lfs.t`

Reviewers: #mercurial, simonfar

Reviewed By: simonfar

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4813193

Signature: t1:4813193:1491212154:0238ea448269b0e5516c927c1fa989d6d9130d4a
2017-04-05 15:49:08 -07:00
Jun Wu
f0b6b81970 lfs: simplify remote blobstore.logic
Summary:
Previously, the remote blobstore could be either `git-lfs` or `dummy`. The
application code does not really care, it only wants a "remote" blobstore.

This diff adds a factory method and makes `git-lfs` and `dummy` stores private.

The `@staticmethod get(vfs)` interface is also removed as it's duplicated and
unnecessary - as long as mercurial calls `reposetup`, the blob store objects
are set, and they cannot be missing.

The error message about an unsupported store is also changed to be consistent
with mercurial style. A test was added to test the error.

Test Plan: `rt test-lfs.t`

Reviewers: #mercurial, simonfar

Reviewed By: simonfar

Subscribers: simonfar, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4813023

Signature: t1:4813023:1491382755:b9d8ec6518141d0ba8263e16c53f430ce80c39f0
2017-04-05 15:48:10 -07:00