Commit Graph

2642 Commits

Author SHA1 Message Date
Durham Goode
248785548b treemanifest: add support for minimal tree pack sending
Summary:
The gettreepack protocol already had the arguments for the client to specify
what nodes they want and what nodes they have, but it wasn't implemented (we
just always returned the entire tree). This patch adds support for only sending
parts of the tree that are new. In addition, if the client requests many nodes,
we will only send the portion of each node that is not contained in the other
nodes. For instance if you ask for X and Y, where X is the parent of Y, then we
will send all of X and only the part of Y that is new.

The order changed a bit, probably because we stopped over delivering data.

Test Plan:
Ran and updated the tests. The next patch will actually add a test
for this code.

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: quark, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4911531

Signature: t1:4911531:1492643897:58aabe5a32f156dd7424cc07c990301dbd5dc1c7
2017-04-19 21:14:04 -07:00
Durham Goode
2565a8cd22 treemanifest: change test to use multiple directories
Summary:
A future patch will make tree prefetching more efficient. Let's change the
prefetch test to have changes in two separate directories so we can observe that
one directory is not downloaded when doing an efficient prefetch.

Test Plan: Ran the test

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4911523

Signature: t1:4911523:1492641009:ac108ebac9573444c87c403f0be0b246d734dda7
2017-04-19 21:14:04 -07:00
Durham Goode
68030f1dd7 treemanifest: add known argument to getancestor api
Summary:
During a repack we often want to access the ancestory for a bunch of nodes that
might be ancestors of each other. Using getancestors for that results in a lot
of duplicated work. For instance, getancestors(0) returns [0], getancestors(1)
returns [0, 1], getancestors(2) returns [0, 1, 2], etc. Which is n^2

This patch adds an optional `known` argument for getancestors that let's the
caller tell getancestors what ancestors it's already aware of. Then getancestors
can short circuit when it reaches that level. This avoids duplicate work during
repack.

Test Plan:
Ran treemanifest repack in our large repo and verified it made
progress over the nodes much faster than before

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4901308

Signature: t1:4901308:1492640896:27d4a90c2993cd1fefbd8dbc211f2ec181178bce
2017-04-19 21:14:04 -07:00
Durham Goode
dae50fc99e treemanifest: support repacking revlogs into packs
Summary:
Accessing treemanifests in revlogs is incredibly slow. This patch adds the
ability for `hg repack` to repack the revlog content into a data pack file, and
teaches the getserverpack code to look in the pack file first.

Test Plan: Need to add a test. Sent out anyway for review

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: quark, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4901298

Signature: t1:4901298:1492640706:2850b9f0e9bbee77952f46af3b784aa81253e626
2017-04-19 21:14:04 -07:00
Durham Goode
42344a282f treemanifest: fix fncache integration
Summary:
Future tests showed that backfilltree was not populating the fncache, which
meant the files were not streamed during a clone or read during a repack. This
fixes this by using the actual repo store (which handles fncache serialization)
instead of a creating a new store.

Test Plan: Added a test

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: quark, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4916070

Signature: t1:4916070:1492647129:4deeb70c63dd414f6d5c06e4259a0bbf899f3425
2017-04-19 21:14:04 -07:00
Durham Goode
3214c9a4df treemanifest: move revlogdatastore to contentstore.py
Summary: This is a generally useful store class. Let's move it to the store code.

Test Plan: Ran the tests

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: quark, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4901293

Signature: t1:4901293:1492647203:0f76a6e78fd0035b61c4e45e18cd9fce59359768
2017-04-19 21:14:04 -07:00
Durham Goode
6a6c59d23d treemanifest: remove n^m behavior in ancestor logic
Summary:
The algorithm did a bfs over the commit graph, but it didn't check if it had
already processed a commit before. This meant every merge ended up traversing
both sides of the merge entirely (even if there was duplicate), and if there was
multiple merges this resulted in n^m behavior.

Test Plan:
Did a treemanifest repack in our big repo and verified it actually
made progress instead of getting stuck in cpu usage for hours

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4901287

Signature: t1:4901287:1492639247:b547e7f4a2051117aff41183ceb78aae44695b7a
2017-04-19 21:14:04 -07:00
Durham Goode
10f743a300 treemanifest: chop off trailing slash when requesting manifests
Summary:
The name being passed to the store was wrong, because it had a trailing slash.
This wasn't caught before because the code was reading and writing the paths
with a slash at the end, so it matched as long as we only interacted with packs
produced by the code. The issue became more obvious when I tried to have packs
generated from revlogs interact with this code.

All the tests are affected since the entry keys changed.

Also use 'const ManifestFetcher &x' to pass the ref to avoid the copy.

Test Plan: Tests updated

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: quark, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4901274

Signature: t1:4901274:1492638476:ff28f8976657baec99effbd82ecd436f6282ea5b
2017-04-19 21:14:04 -07:00
Durham Goode
b30215f840 treemanifest: change server to use ctreemanifest for reading revlogs
Summary:
The server treemanifest reading code was super slow because it used the pure
python treemanifest implementation. Let's switch it to use the ctreemanifest
implementation on top of a revlog store class.

The tests change because the output is now in reverse topological order (deepest
trees first).

Test Plan: Tests updated

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4901268

Signature: t1:4901268:1492637601:371a2303ac4f6e2b81f69a85ca657849150abcf7
2017-04-19 21:14:04 -07:00
Durham Goode
45a47ec954 treemanifest: fix serialization during walksubtree of non-finalized trees
Summary:
Previously, walksubtree would segfault if you ran it on a tree that had
non-finalized entries, since they contain node pointers that point to NULL.
Let's fix this by having serialization refer to the nullid if the pointer is
null.

Test Plan: Updated the tests to cover this

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4901257

Signature: t1:4901257:1492637151:803b3183ad766e34f0ad24f14a8ec1b784600879
2017-04-19 21:14:04 -07:00
Durham Goode
e1744f2469 treemanifest: remove accidental use of '\0' * 40
Summary:
In one place we tried to set the node to null ('\0' * 20) but instead set it to
40 null bytes. If we're using a hex representation we should be using '0' * 40.
Let's make a global constant for this as well.

Test Plan: Ran the tests

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4901253

Signature: t1:4901253:1492637015:15bca25bff67167c3c1f51179b7c6f4db2589586
2017-04-19 21:14:04 -07:00
Durham Goode
b16fef8f08 treemanifest: add ManifestEntry.hasNode
Summary:
Previously, callers would need to be aware of how ManifestEntry stored its
no-node state, by checking if entry.node == NULL. Moving to a hasNode() api
allows us to be more flexible in how we represent an empty node.

Test Plan: Ran the tests

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4901246

Signature: t1:4901246:1492637096:bc193dd998d9f9653dd17b3324a2e71ac8732235
2017-04-19 21:14:04 -07:00
Durham Goode
c0c997fb89 treemanifest: add walksubtrees api
Summary:
Adds a walksubtrees api that returns a set of subtrees from the given
treemanifest. This will be used in a later patch to return just the subtrees
that are new in one treemanifest. For now it's only used by the unit tests

Test Plan: Added unit test

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: quark, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4901241

Signature: t1:4901241:1492645063:5b3617cd36f3205bef8e09c4d37dd132e0bece4e
2017-04-19 21:14:03 -07:00
Durham Goode
e0d844028e treemanifest: refactor finalizing logic out of subtreeiter
Summary:
A future patch will want to use SubtreeIterator for comparing two arbitrary
trees. To do this, we need to refactor out all the logic that actually computes
permanent hashes based on the other tree (since if I'm comparing against an
arbitrary tree, I don't want to use its hashes as my p1/p2 like I would if I'm
serializing the new parts of a tree).

This drastically simplifies the code as well, making it much easier to maintain.

Test Plan: Ran the tests

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4895795

Signature: t1:4895795:1492629678:18617385720b0e1ef5d1513085910a0d3968e29b
2017-04-19 21:14:03 -07:00
Durham Goode
f51d0f1465 treemanifest: add FinalizerIterator
Summary:
A future patch will make SubtreeIterator more generic for diffing trees. To do
that we'll need to move the finalization logic out of SubtreeIterator. This
patch adds a new FinalizerIterator class that simply wraps SubtreeIterator. In
the next patch logic will move between the two classes.

Test Plan: Ran the tests

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4895794

Signature: t1:4895794:1492629514:9ee6e60c32e99041ce16f92c2923de6b4ad0b104
2017-04-19 21:14:03 -07:00
Durham Goode
d7d340e50e treemanifest: add updatebinnode and updatehexnode
Summary:
A future patch will call ManifestEntry.update in more places, and it is
convienent to have the bin-to-hex conversion unified into one place.

Test Plan: Ran the tests

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4895788

Signature: t1:4895788:1492629387:7d64db0169bc47e5996666c4fb8fde639e83ba8b
2017-04-19 21:14:03 -07:00
Durham Goode
05f39b75e9 treemanifest: add test for no-op subtree change
Summary:
Adds a test for when a tree is modified, then modified back to the original
content. When serializing, it should not result in new trees for that directory,
but should result in a new root tree since p1 has changed.

Test Plan: Ran the test

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: quark, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4880087

Signature: t1:4880087:1492629218:93de40aede28b6b0f5b02e7015d96c61e531cc6a
2017-04-19 21:14:03 -07:00
Durham Goode
f706fa0de0 treemanifest: move manifest node onto Manifest
Summary:
Previously the manifest node was only contained by the pointer to the manifest
(usually the container manifest). This made accessing the node awkward in a
bunch of places. Let's refactor it so the node is kept on the Manifest itself as
well, and it handles invalidation and computation of that node as it is editted.

Test Plan: Ran the tests

Reviewers: #mercurial, stash

Reviewed By: stash

Subscribers: stash, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4880076

Signature: t1:4880076:1492591856:f8cd065cac8d3f7cac3069ee9cb61881a24ee426
2017-04-19 21:14:03 -07:00
Durham Goode
b0e5a8d613 treemanifest: add exception handling to newtreeiter.next
Summary:
While debugging I noticed newtreeiter_iternext doesn't handle c++ exceptions.
Let's handle it so the process doesn't crash entirely.

Test Plan:
Ran the tests.  This kind of exception shouldn't be possible in
normal execution, so there's not really a test to write.

Reviewers: #mercurial, stash

Reviewed By: stash

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4880068

Signature: t1:4880068:1492590284:87f986dc9fa3650524d8c71f69c28971e2552640
2017-04-19 21:14:03 -07:00
Durham Goode
f2405cda64 treemanifest: update finalize tests
Summary:
The current tests test some undefined behavior of manifest.finalize() where it
can rewrite immutable Manifest instance's with a new node. This happens if you
call finalize without passing the appropriate parent that the manifest was
copied from. So the new tree contains unmodified subtrees from the original
manifest, and since we don't pass the original manifest to finalize, it doesn't
realize these trees already existed and tries to give them a new node.

A future patch makes tree mutability more strict and caught this. Fixing the
tests now, so I can show that future patches don't break this.

Test Plan: Ran the test

Reviewers: #mercurial, stash

Reviewed By: stash

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4880057

Signature: t1:4880057:1492589057:a7026bc791ea2cd12aa790513cf46d5180cb0638
2017-04-19 21:14:03 -07:00
Durham Goode
c1f233a304 treemanifest: rename NewTreeIterator to SubtreeIterator
Summary:
The NewTreeIterator class was used for iterating over a new tree, when given
it's p1. Mainly for performing the write of a new tree. We want to make this a
more generic mechanism so we can do the diff of one tree with an arbitrary
nother (like when deciding what trees to send a client if they have a given base
node). Let's start by renaming the class.

Test Plan: Ran the tests

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4880048

Signature: t1:4880048:1492110564:bd51d35e4cfcca2d6b1d4cf81d78fdacf2b52f3d
2017-04-19 21:14:03 -07:00
Durham Goode
8da9d7b37f treemanifest: automatically download missing trees on demand
Summary:
This adds a remote tree datastore to the store, so if we don't have a tree on
local disk, we can reach out to the server for it.

This also makes a small change to fastmanifest to use get() instead of
getmissing() to test if the tree is available. getmissing() would indicate the
tree is missing if it wasn't local, while get() will attempt to fetch it from
the server.

Test Plan: Added a test

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: quark, stash, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4864645

Signature: t1:4864645:1492627823:7afff5db54c895507507e57ab7263f4dbf10b5f7
2017-04-19 21:14:03 -07:00
Durham Goode
b44263e197 fixtests: move treemanifest configs onto client only
Summary:
The treemanifest tests were setting the configs on both the client and the
server, without telling the server it was a server and without setting it up
appropriately to be a client. This caused the tests to fail in a future patch.
Let's move the treemanifest config to the clients, since that's the only bits
being tested here.

Test Plan: Ran the tests

Reviewers: #mercurial, ikostia

Reviewed By: ikostia

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4864766

Signature: t1:4864766:1491903651:ab91eed5a2eb8a60db8e230c2ba5c0ee29b5adc9
2017-04-19 21:14:03 -07:00
Durham Goode
3633627260 treemanifest: add prefetchtrees for receiving trees from the server
Summary:
This adds a new prefetchtrees command and gettrees wireprotocol command for
receiving trees from the server. The actual wire protocol is a custom request
format, with a normal pack file response format (same as used in remotefilelog).
The request args can specify which manifest nodes are wanted, for which
directory and which subdirectories.

I attempted to use the changegroup3 response format instead of the pack format,
but changegroup is extremely intertwined with revlogs and with producing
changegroups for linear swaths of the repo history, instead of random parts of
just the manifest.

Test Plan: Added a test

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: stash, quark, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4864435

Signature: t1:4864435:1492626392:c3603eeb261a1f9b9d836f0d44614025b58d45ea
2017-04-19 21:14:03 -07:00
Arun Kulshreshtha
02e5cc57ba tweakdefaults: allow hg update --merge without --nocheck
Summary: The --merge flag is relatively new; we shouldn't add --check if it is specified.

Test Plan: Unit test

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4916396

Tasks: 17490374

Signature: t1:4916396:1492647440:d9955931cb9177719a58325c08a715d4d51d1226
2017-04-19 17:20:23 -07:00
Arun Kulshreshtha
6065ceef3f fbamend: use unfiltered repo when clearing preamend bookmarks
Summary: Sometimes restack attempts to clear preamend bookmarks on hidden commits. This only happens when the user has a configuration error (typically inhibit is disabled) which would mess up other parts of restack. Nonetheless, there is no reason to crash here.

Test Plan: Disable inhibit, perform an amend, and run `hg restack`. Unit test is forthcoming.

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4856640

Tasks: 16726367

Signature: t1:4856640:1491816803:56beeb11360a7a2f6bc6243069c562b08a015afc
2017-04-19 15:52:45 -07:00
Wez Furlong
04ee6433d4 warnings: squelch contextlib.nested DeprecationWarnings
Summary:
deals with the tests failing like this on my python 2.7 system:

```
+  /data/users/wez/facebook-hg-rpms/fb-hgext/hgext3rd/uncommit.py:144: DeprecationWarning: With-statements now directly support multiple context managers
+    with nested(repo.wlock(), repo.lock()):
```

Test Plan: arc unit

Reviewers: #mercurial, stash, simonfar, wez

Reviewed By: stash, simonfar

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4910040

Signature: t1:4910040:1492593520:ff815c5dbed4a341e0a313cd7160b700d878ee2c
2017-04-19 03:28:23 -07:00
Simon Farnsworth
e31246ce99 rage: include klist output
Summary:
We're seeing a rise in the number of auth issues. As a first step,
let's have rage grab klist, so we can see if the user has a valid Kerberos
ticket for hg.vvv or not.

Test Plan: Run it and see https://phabricator.intern.facebook.com/P57315650

Reviewers: #sourcecontrol, ikostia

Reviewed By: ikostia

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4907615

Signature: t1:4907615:1492546856:969cab7b8deacf220c374d4c9e1b6e99f84ca059
2017-04-19 03:18:04 -07:00
Kostia Balytskyi
aeef0ad5cc compatibility: migrate from scmutil.vfs to mercurial.vfs.vfs
Differential Revision: https://phabricator.intern.facebook.com/D4908906
2017-04-18 14:42:33 -07:00
Wez Furlong
934dfaf08b templater: fixup for templater changes upstream
Summary:
The function signature for a number of functions relating to templating changed.

In addition, the templater layer now requires that template functions have been
registered via the decorator, otherwise a require attribute is missing and
things blow up.

Test Plan: arc unit.  rt.

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4908220

Signature: t1:4908220:1492545033:66f089e03a6064ae96ca36f3abf99ecb6d48d422
2017-04-18 13:04:11 -07:00
Wez Furlong
7170cd6411 tweakdefaults: switch hg grep to use dirstate.walk
Summary:
previously, this was using .match() to find candidate names
and then reaching inside the dirstate._map to check the status of
the file.

This implementation uses the dirstate.walk instead, which returns a
map of filename -> stat information.

This approach works better with custom dirstate implementations, such
as eden, where a local map of the required information is not available
and where it would be suboptimal to request that information on
demand on a per-file basis.

Test Plan:
testing before and after this change:

`time hg grep eden foo` has a runtime of ~1.2 seconds.

If you have sqldirstate enabled, this change makes the runtime 3 seconds
compared to 2 with sqldirstate on and without this change.

Reviewers: #mercurial, durham

Reviewed By: durham

Subscribers: mjpieters, simpkins

Differential Revision: https://phabricator.intern.facebook.com/D4903110

Signature: t1:4903110:1492473495:ebe375f3e2533f6ae4ca7ea9a494f6feb9704f9e
2017-04-18 10:05:31 -07:00
Stanislau Hlebik
928a96b602 infinitepush: use ui.debug() instead of ui.warn()
Summary: To avoid spamming users

Test Plan: Run tests

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4883447

Signature: t1:4883447:1492088221:3649b14ff514957d2f412aae2647ca1b20ef5fea
2017-04-17 23:44:26 -07:00
David Soria Parra
61c9ea8ec7 p4fastimport: require p4 during tests
Summary: Require perforce during all tests.

Test Plan:
run tests with and without Perforce in PATH. Tests correctly run with
P4D and P4 in path and were correctly skipped without.

Reviewers: #sourcecontrol, #idi, durham

Reviewed By: durham

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4899085

Signature: t1:4899085:1492475858:3bd1443c707e56461835d278a9c6bf3e034b5f4a
2017-04-17 18:08:07 -07:00
Arun Kulshreshtha
a72a47d515 absorb: s/chunks/chunk
Summary: Noticed this minor gramatical error when using absorb.

Test Plan: Unit tests.

Reviewers: quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4887040

Signature: t1:4887040:1492123047:dcae53cabd64f1df13d80a20953c712ce0aaaa09
2017-04-13 16:00:51 -07:00
David Soria Parra
ef08c10f5b p4fastimport : introducing fast Perforce to Mercurial convert extension
Summary:
`p4fastimport` is a fast convert extensions for Perforce to Mercurial. It
is designed to generate filelogs in parallel from Perforce. It tries to
minimize the use of Perforce commands and reads from the the Perforce
store on a Perforce server directly.

The core of p4fastimport is the idea to generate a Mercurial filelog
directly from the underlying Perforce data, as a Perforce file in most
cases matches a filelog directly (per-file branches is an exception). To
generate a filelog we are reading each file for an imported revision. A
file in Perforce is locally either stored in RCS, as a compressed GZIP
or as an flat file (binaries). If we do not find a version locally on
disk we fallback to downloading it from Perforce.

We are generating manifests after all filelogs are imported. A manifest
is constructed by adding and removing files from an initial state. We
are generating the correct offset from a manifest into the filelog by
keeping track of how often a file was touched.

We then generate the changelog.

Linkrev generation is a bit tricky. For every file in Perforce know
to which changelist it belongs, as it's stored revisions contains the
changelist. E.g.  1.1422 is the file changed in the changelist 1422 (this
refers to the "original" changelist, before a potential renumbering,
which is why we use the -O switch).  We use the CL number obtained
from the revision to reverse lookup the offset in the sorted list of
changelists, which corresponds to it's place in the changelog later,
and therefore it's correct linkrev.

Parallel imports: In order to run parallel imports we MUST keep one lock
at a time, even if we import multiple file logs at the same time. However
filelogs use a singular `fncache`, which will be corrupted if we generate
filelogs in parallel. To avoid this, repositories must be generated with
*fncache* disabled! This restricts `p4fastimport` with workers to run
only on case sensitive file systems.

Test Plan:
The included tests as well as multiple imports from a small testing
Perforce client. Afterwards successfully run `hg verify`

  make tests

Reviewers: #idi, quark, durham

Reviewed By: durham

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4776651

Signature: t1:4776651:1492015012:0161c4f45eab4d3b64597d012188c5f2007e8f7d
2017-04-13 11:11:09 -07:00
Stanislau Hlebik
57dc185f42 infinitepush: don't make separate backups for different working copies
Summary:
Previously we had separate backup per working copy. That's very confusing
since all these working copies shares the same repo. This diff fixes it and
also adds config option to clean unnecessary working copy server-side.

Test Plan: Run infinitepush unittest

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: rmcelroy, quark, mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4876230

Signature: t1:4876230:1492025747:3579e5046efc2ed309044fc3335c36ac4f7bdd04
2017-04-13 04:58:11 -07:00
Wez Furlong
4a59f3b701 c-extensions: fixup some compiler/environment portability concerns
Summary:
I sync'd a copy of this code into the eden repository.
I had to adjust a couple of include paths to get the code to
compile correctly in the hermetic build environment that is
in use there.

In addition, our linter suite over there found a couple of C++ nits
to be fixed up.

Test Plan: make local

Reviewers: simpkins, ikostia, simonfar, durham

Reviewed By: durham

Subscribers: net-systems-diffs@fb.com, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4879285

Signature: t1:4879285:1492039044:8cb1e033e35ee568806de94dda3d2f6f8e78f5cb
2017-04-12 16:34:53 -07:00
Adam Simpkins
642dea7e2c fixcorrupt: automatically look back as far as necessary
Summary:
Update the fixcorrupt extension to look back as far as necessary in the revlog
to find a good entry.  Previously the user had to supply a good value using the
--checklen argument.  If --checklen was too small, fixcorrupt could end up
trying to truncate in the middle of a corrupted section of the revlog, which
would cause problems as it was still using corrupted data.

This change ensures that fixcorrupt always looks back far enough to find a good
entry.  We start by looking at the last 10 entries, and double that amount each
time the first item in the list is still bad.

Test Plan:
Used it to fix a corrupted fbsource repository where the corruption was 36
commits back.

Reviewers: #mercurial, quark, stash

Reviewed By: stash

Subscribers: stash, net-systems-diffs@fb.com, yogeshwer, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4874243

Signature: t1:4874243:1492007518:7406b03a9967815a496a8c2fae394c5f21f7e60a
2017-04-12 14:09:22 -07:00
Durham Goode
4da0040f03 fastlog: fix ScmQuery request parameter name
Summary:
This was totally wrong and has been forever. I think we got lucky because the
server was serving from memcache mostly, which did not validate this parameter.

Test Plan:
Ran log before and after the change.

hg.real log wdc --config extensions.fastlog=../../facebook-hg-rpms/fb-hgext/hgext3rd/fastlog.py

Reviewers: #mercurial, mitrandir

Reviewed By: mitrandir

Subscribers: mjpieters, jeroenv

Differential Revision: https://phabricator.intern.facebook.com/D4878633

Tasks: 17327887

Signature: t1:4878633:1492030224:cf1b802345c7e00d1134ee81fbe9f271cf1a6752
2017-04-12 13:51:18 -07:00
Mateusz Kwapich
75778a9b81 remotefilelog: fix tests
Summary: Only one exact form on "hg" invocation with --stdio is now allowed

Test Plan: tests are passing now

Reviewers: #mercurial, simonfar

Reviewed By: simonfar

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4877162
2017-04-12 13:21:16 -07:00
Stanislau Hlebik
8f8cdc2d49 hgext3rd: cleanobsstore extension
Summary: See comments and docstring

Test Plan: arc unit

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: quark, mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4851138

Signature: t1:4851138:1491589247:5ca750e45bcc9fc996c9d7c14e31cb5f24ea983f
2017-04-12 01:16:01 -07:00
Stanislau Hlebik
efbec7652d infinitepush: fix security concern
Summary: Check owner of a log file before writing to it. See comments for details

Test Plan: arc unit

Reviewers: #mercurial, simpkins

Reviewed By: simpkins

Subscribers: net-systems-diffs@fb.com, mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4842933

Tasks: 17155924

Signature: t1:4842933:1491861137:e8a027fb7a930c0c5f553c75cb84214d24f66ce3
2017-04-12 01:12:23 -07:00
Jun Wu
e0eb41504b test-fixcorrupt: be OSX compatible
Summary:
`wc` in OSX has different padding behavior that breaks the test:

```
       $ wc -c .hg/store/00changelog* .hg/store/00manifest* | sort
    -   99 .hg/store/00manifest.d
    -  110 .hg/store/00changelog.d
    -  128 .hg/store/00changelog.i
    -  128 .hg/store/00manifest.i
    -  465 total
    +        99 .hg/store/00manifest.d
    +       110 .hg/store/00changelog.d
    +       128 .hg/store/00changelog.i
    +       128 .hg/store/00manifest.i
    +       465 total
```

This patch fixes it.

Test Plan: The fix was verified manually on an OS X machine.

Reviewers: #mercurial, durham

Reviewed By: durham

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4872689

Signature: t1:4872689:1491954172:92178178deea88ec3545c42ad0cdecc8099a1db6
2017-04-11 16:43:32 -07:00
Jun Wu
340750c4f4 pushrebase: send obsmarkers to client if client supports it
Summary:
Previously when the client has obsstore enabled, and the server has obsstore
disabled, clients won't get the obsmarkers, which is suboptimal. This diff
makes the server send obsmarkers in that case.

Practically, this means people will no longer need to run `strip` after landing
a diff in this repo.

Note: `test-pushrebase-manifests.t` is somehow flaky about the empty lines. I
guess they are not related to the change. But the test change is not 100%
reproducible.

Test Plan: Added a new test case

Reviewers: #mercurial, stash

Reviewed By: stash

Subscribers: rmcelroy, ikostia, stash, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4865150

Signature: t1:4865150:1491895844:afbc3079a40a9a9fa9af1eab4eeaca91091e8d2d
2017-04-11 13:21:27 -07:00
Jun Wu
11cd7af681 lfs: add a fast path for filelog.size
Summary:
Previously, `filelog.size` requires loading the lfs blob in memory. This
makes it unnecessary.

Test Plan: Run existing tests.

Reviewers: #mercurial, mitrandir

Reviewed By: mitrandir

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4861358

Signature: t1:4861358:1491923349:f1d2fe5656158854e1dd50987ea3db64a9793db6
2017-04-11 13:20:58 -07:00
Jun Wu
de5e7c6545 fixcorrupt: new extension to fix corrupted repo
Summary:
We sometimes get reports about corrupted repos. Usually the corrupted part
is just at the end of changelog or manifest.

Truncating them manually works but people need to be very careful. This
extension is like the manual fix but automatized.

Test Plan:
Run `hg debugfixcorrupt --no-dryrun` on a reported corrupted repo and check
it truncates files correctly and the repo passes `hg verify` afterwards.

Reviewers: #mercurial, durham, stash

Reviewed By: stash

Subscribers: stash, rmcelroy, durham, lcharignon, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D3408396

Signature: t1:3408396:1491897232:fc17a105124b568963441adfec97e26735df3258
2017-04-11 13:19:42 -07:00
Adam Simpkins
fdc2c3e6f9 extutil: add unit tests for runbgcommand
Summary:
Add some basic unit tests for extutil.runbgcommand().

This also changes the behavior to throw an OSError if we fail to execute the
process, instead of a subprocess.CalledProcessError.  This matches the behavior
of the subprocess module when it fails to execute the command.

Test Plan: Ran the tests.

Reviewers: stash, quark

Reviewed By: quark

Subscribers: net-systems-diffs@fb.com, yogeshwer, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4864999

Signature: t1:4864999:1491874918:c03edafe02af217e41c28a770137bfd72bcbba9b
2017-04-11 11:25:40 -07:00
Adam Simpkins
05c70db64f build: make sure portability/ is always added to the include path
Summary:
The portability/ subdirectory always needs to be listed in the compiler include
paths in order to successfully build.  This fixes setup.py to always add it to
the list of include directories, rather than only adding it when INCLUDE_DIRS
was not specified through the environment.

Test Plan:
Successfully built the fb-hgext repository when specifying alternate system
header paths through the INCLUDE_DIRS environment variable.

Reviewers: tja, wez, ikostia

Reviewed By: ikostia

Subscribers: net-systems-diffs@fb.com, yogeshwer, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4865296

Signature: t1:4865296:1491903269:fc07edf01a89117e25805e9ee936b2e2eee22410
2017-04-11 11:17:05 -07:00
Ryan McElroy
8267123b73 scm-prompt: add eden snapshots support
Summary:
Based on the recent commit wez made, but centralized to the scm-prompt
source-of-truth.

Test Plan: updated test

Reviewers: #sourcecontrol, wez, simpkins, simonfar

Reviewed By: simonfar

Subscribers: net-systems-diffs@fb.com, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4867662

Tasks: 17110799

Signature: t1:4867662:1491916786:647edb819a059a3a6baa09c8d189ea706933687b
2017-04-11 08:47:12 -07:00
Pallav Shinghal
3ded80a5c7 rage: fix hg sparse entry in hg rage output
Summary: The `hg sparse` entry in `hg rage` output is consistently `(Failed. See footnote [1])` due to the `sparse` function being called with the wrong signature.

Test Plan:
Ran `hg rage` in <reponame>. Output contained:

```
hg sparse:
---------------------------
%include .hgsparse-<reponame>
[include]

[exclude]

```

Reviewers: #sourcecontrol, quark

Reviewed By: quark

Subscribers: quark, mjpieters, dawidp

Differential Revision: https://phabricator.intern.facebook.com/D4861832

Tasks: 17269067

Signature: t1:4861832:1491869628:0abc55eb5308a83effe78943dba8d13e9f12d616
2017-04-11 07:19:26 -07:00