Commit Graph

3067 Commits

Author SHA1 Message Date
Jun Wu
3d56581781 lfs: store isbinary information in LFS metadata
Summary:
Usually LFS files are binary files. But there could be exceptions. This diff
adds a new customized field `x-is-binary` to record those exceptions.

The `filectx.isbinary` API is changed to use that metadata as a fast path.

This allows us to provide a transparent user experience (whether a file is
stored in LFS or not does not affect its original `isbinary` property),
while still being able to skip loading the LFS blob if the file is binary.

Test Plan: Added a new test case

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D5001117

Signature: t1:5001117:1493892406:2a8ebd42d0ae0dbae39b87b9ea16db22b99f3d12
2017-05-04 09:18:04 -07:00
Martijn Pieters
372c6c955f sparse: restore sparse profile on exception during import
Summary: Like _config, _import needs to restore the pre-existing profile if refreshing failed. Otherwise we have marked new rules as imported and applying without that actually being true.

Test Plan: rt test-sparse*

Reviewers: #sourcecontrol, rmcelroy, durham, simonfar

Reviewed By: simonfar

Subscribers: simonfar, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4985415

Tasks: 17793415

Tags: autosparse

Signature: t1:4985415:1493893462:ee4bcbf558706c3d2f58367dd7bf2533ccf1b824
2017-05-04 04:58:43 -07:00
Jun Wu
4131d4ebcd pushrebase: fix a typo in comment 2017-05-03 16:19:11 -07:00
Jun Wu
d82b8b1495 pushrebase: do not require exchange for getting markers
Summary:
D4865150 and D4934720 aren't effective in our current setup. The direct
cause in the code is because the server couldn't find common marker version:

```
  # old server-side code, returns empty in our current setup
  obsolete.commonversion(bundle2.obsmarkersversion(reply.capabilities))
```

Upon investigation, it's because there is no `exchange` enabled client-side.

But we do want one-way (server->client) markers for the rebased commits, as
long as obsstore is enabled (createmarkers is set, without exchange).

The upstream expects the server to have obsstore enabled, and exchange
enabled, to send markers. Since we are generating markers without an
obsstore (see D4865150), we are on our own way. This diff makes it one step
further.

This diff adds an explicit parameter to the `b2x:rebase` part to tell the
server what obsmarker format the client supports so the server could make a
right decision without relying on the "standard" `reply.capabilities`, which
is affected by the exchange option.

Test Plan: Change the existing test, make sure the old code fails.

Reviewers: #mercurial, durham

Reviewed By: durham

Subscribers: durham, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4997972

Signature: t1:4997972:1493848751:14c29654b2e8246bd12a8de8820af5b3773e2fb7
2017-05-03 16:08:35 -07:00
Jun Wu
d3f8c4e9dc lfs: override filelog.renamed code path
Summary:
An `hg pull` test triggers the following code path server-side when using
remotefilelog:

  ...
  remote:   File "/usr/lib64/python2.7/site-packages/remotefilelog/remotefilelogserver.py", line 308, in streamer
  remote:     text = _loadfileblob(repo, cachepath, path, node)
  remote:   File "/usr/lib64/python2.7/site-packages/remotefilelog/remotefilelogserver.py", line 223, in _loadfileblob
  remote:     text = createfileblob(filectx)
  remote:   File "/usr/lib64/python2.7/site-packages/remotefilelog/remotefilelogserver.py", line 348, in createfileblob
  remote:     ancestors.extend([f for f in filectx.ancestors()])
  remote:   File "/usr/lib64/python2.7/site-packages/mercurial/context.py", line 1072, in ancestors
  remote:     for parent in c.parents()[:cut]:
  remote:   File "/usr/lib64/python2.7/site-packages/mercurial/context.py", line 923, in parents
  remote:     r = fl.renamed(self._filenode)
  remote:   File "/usr/lib64/python2.7/site-packages/mercurial/filelog.py", line 62, in renamed
  remote:     t = self.revision(node)
  ...

That triggers downloading a blob. We don't want to do that server-side. So
override the `renamed` method to use LFS fast path to test rename.

Practically, this reverts part of D4906074.

Test Plan: Run existing tests. This change was made on the server.

Reviewers: davidsp, #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4992421

Signature: t1:4992421:1493802628:2bf2cf819bfed2aa61ea1c2323c03ab428732815
2017-05-03 11:10:48 -07:00
Augie Fackler
f198dbbc1e morestatus: add config knob to allow some states to be skipped
The main rationale for this change is that I've got years of use
patterns on bisect where I *start* bisect by running 'hg bisect
--reset', not *finish* with that command, so `hg status` with
morestatus enabled is spewing annoyances in nearly every repository
I've ever used. It wasn't any harder to allow people to ignore
anything they want, but (at least for now) bisect is the only state
that seems "special" in that it doesn't have a definite "end" that
Mercurial can detect and automatically clean up. I'd also be fine with
adding something like morestatus.ignorebisect if we want to avoid a
potential footgun where a user could ignore rebase state.
2017-05-03 13:34:36 -04:00
Durham Goode
0723212833 histpack: switch getnodeinfo to use common _findnode function
Summary:
It should've probably been using this function from the beginning, I just
forgot. Let's switch it to use _findnode so it takes advantage of the new
indexes.

This changes the return value of _findnode, but that's fine because the only
caller was getmissing() which didn't use the return value.

Test Plan: Ran the tests. The getnodeinfo unit test covers this code path.

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4991411

Signature: t1:4991411:1493810207:7f694d41089a6efab822ae002645ed1531f6b344
2017-05-03 10:19:46 -07:00
Durham Goode
51cc08d30b histpack: update unit tests to use version 1
Summary:
The tests were still testing version 0. Let's update it to version 1 and fix the
bad version check test.

Test Plan: Ran the tests

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4991404

Signature: t1:4991404:1493802873:def4135219d731f7c0cb542012e439e79192c54f
2017-05-03 10:19:46 -07:00
Durham Goode
980bf7ecdb histpack: switch _findnode to use node index
Summary:
Now that we have node indexes, let's switch _findnode (which is responsible for
random access node lookups) to use that index.

Test Plan: Ran the tests. The getmissing unit test covers this code path.

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4991400

Signature: t1:4991400:1493802781:6422be2522c3a423c80e41ce776057720b32ee98
2017-05-03 10:19:46 -07:00
Durham Goode
ef4d4cca3a histpack: move bisect to it's own function
Summary:
A future patch will be doing a similar bisect over a different index, so let's
move the bisect logic to it's own function.

Test Plan: Ran the tests

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4991393

Signature: t1:4991393:1493802679:1fb20c1e1ae6271f10953889f9d4d524d3e03a76
2017-05-03 10:19:46 -07:00
Durham Goode
bbebdda760 histpack: move entry reading to a function
Summary:
A future patch will be doing entry reading from another code path, so let's
refactor it out.

Test Plan: Ran the tests

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4991388

Signature: t1:4991388:1493802655:c20b3e394a785b31c8e94cfb0285575c6ceed70d
2017-05-03 10:19:45 -07:00
Durham Goode
b559010043 treemanifest: refactor _findnode
Summary:
Now that _getancestors is an iterator, we can get rid of the custom scan code in
_findnode.

Test Plan: Ran the tests

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4991386

Signature: t1:4991386:1493798953:e3f7faa828ef3148f8c66930d04c20c6fa83b83b
2017-05-03 10:19:45 -07:00
Durham Goode
16b8b169b7 treemanifest: return node index from _findsection
Summary:
Now that we have node indexes, let's return their location from _findsection.
Patch includes a slight rename to a constant that was poorly named as well.

Test Plan: Ran the tests

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4991385

Signature: t1:4991385:1493798906:c3147d00b6fc67b47191592032519afccf71e9af
2017-05-03 10:19:45 -07:00
Durham Goode
96ea7cd405 histpack: add per-node index
Summary:
Previously looking up a particular node in a histpack required a bisect to find
the file section, then a linear scan to find the particular node.  If you needed
to look up the latest 3000 nodes one by one, that involved 3000 linear scans,
many of which traversed the same nodes over and over.

This patch adds additional index at the end of the current histidx file. In a
future patch, we will change getnodeinfo() to use this index instead of the
linear scan logic.

Test Plan:
Ran the tests. I haven't actually verified that the data in these
indexes is correct. My next patch will add logic that reads these indexes and
will add tests around it. I won't land this until I've confirmed it's correct.

Reviewers: quark, #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4983690

Signature: t1:4983690:1493798877:ae0802b4896b54bf066df9f684d94554855fd35a
2017-05-03 10:19:45 -07:00
Durham Goode
d1a927d335 packs: add entry count to pack index
Summary:
Previously, we used the length of the index file to determine the upper bounds
of the bisect. In a future patch we'll want to add more data to the end of the
index file, so we need to record how long the index portion of the index is.
This patch adds that information.

Test Plan: Ran the tests.

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4983682

Signature: t1:4983682:1493693255:57ab9af2030847fedff05b6755113ba8ce0c933b
2017-05-03 10:19:45 -07:00
Durham Goode
8674a90bb5 histpack: add version 1
Summary:
This patch just bumps the histpack version number to 1 and adds a config flag to
enable writing v1 pack files. The format hasn't actually changed in this patch,
I'm just doing the verison bump so I can update all the hashes in the tests
without working about functionality change.

In the next patch I will modify the index format, which won't affect the hashes.

Test Plan:
Ran the tests. I also ran the tests with some debug code to manually
force the sha to include 0 instead of 1 and verified that the hash didn't change
(which confirms that all of these hash changes are just because of that one byte
version change).

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4983675

Signature: t1:4983675:1493692444:5d88df4d46ce487f1b791417754ba000ecf10a1e
2017-05-03 10:19:45 -07:00
Stanislau Hlebik
8b3bb637b4 infinitepush: fix skipping downloading bundle
Summary:
`bundlefile` variable wasn't initialized if `head in nodestobundle` were true.
Even worse, the previous version of `bundlefile` were used. That means that in
some cases infinitepush sends back a wrong bundle.

Adding a unittest is quite tricky because problem shows up only in one rare
case. Example: we have a stack of 2 commits A and B and each commit in the
stack has a bookmark that points to it. Then `getbundlechunks()` `heads`
parameter contains two heads: commit A and commit B in undefined order.
If it happens that first element in the list is commit A we'll create a bundle
that contains only commit A. And we'll reuse the same bundle when we process
commit B. That means that we'll send the same bundle with commit A twice,
but no bundles with commit B.

I refactored the code to make sure we won't get the same bug in
the future.

Test Plan: arc unit

Reviewers: #mercurial, durham

Reviewed By: durham

Subscribers: durham, mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4969547

Tasks: 17719083, 15389402

Signature: t1:4969547:1493749313:587f9e4446a3c21b47c081a0fe4cd9e200dab5cd
2017-05-03 00:31:24 -07:00
David Soria Parra
168184f996 p4fastimport: use email + name from user database
Summary:
Use the perforce database to construct a "User <Email>" author
correctly.

Test Plan: rt test-p4* test-check*

Reviewers: #idi, #sourcecontrol, wlis

Reviewed By: wlis

Subscribers: wlis, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4986521

Signature: t1:4986521:1493755274:cb41e8f6661ce76c54cd32cfe70bc4372e1e2161
2017-05-02 15:43:39 -07:00
David Soria Parra
01ddd1a4e3 p4fastimport: remove unused p2 variable
Summary: remove unused p2 variable

Test Plan: rt test-*

Reviewers: #sourcecontrol, quark, durham

Reviewed By: durham

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4985735

Signature: t1:4985735:1493764630:fd0830e1f5a606983fc03ca5b34732c77b19ed3a
2017-05-02 15:43:39 -07:00
Durham Goode
e1f036c158 treemanifest: write local trees during finalizer phase
Summary:
Previously we wrote the local trees (during hg commit) in the postclose part of
a transaction, so we could actually fail to write the trees but the transaction
was already successful. Let's move this to the finalizer phase instead.

This also means other services can depend on the trees existing by the time post
close hooks are called.

Test Plan:
Ran the tests. Verified dependent services could now see the packs
during postclose.

Reviewers: #mercurial, wez

Reviewed By: wez

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4989624

Signature: t1:4989624:1493763114:dcb00adadd62c1d2fc5b0254eb1593ebc8572f85
2017-05-02 15:12:18 -07:00
Jun Wu
488cf5ea26 cdatapack: fix empty revision reading
Summary:
The newly changed `LZ4_decompress_safe` is unhappy about empty buffer. So let's
check empty revision explicitly.

Test Plan: Added a test

Reviewers: #mercurial, ikostia

Reviewed By: ikostia

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4986277

Signature: t1:4986277:1493743735:be2ad6047bb0b983425c6e49b7c5ddf2c94d1c1a
2017-05-02 10:01:13 -07:00
Jun Wu
004e19698b setup: add a command to remove extra build files
Summary:
I noticed that cdatapack.o was not cleaned with `make clean`. And `.c`
generated by Cython should also be cleaned to avoid surprise.

Since the latter cannot be done cleanly using short shell script, write the
clean logic in Python.

Test Plan: `make clean`

Reviewers: #sourcecontrol, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4979676

Signature: t1:4979676:1493717290:a826da9e50bcd67362c790b27ff0d88714d71d49
2017-05-02 09:12:56 -07:00
Martijn Pieters
3651652e57 remotefilelog: move test to tests directory
Summary:
This test won't be run unless it is in the right directory.

#m brownpaperbag

Test Plan: run arc unit

Reviewers: #sourcecontrol, jeroenv

Reviewed By: jeroenv

Subscribers: jeroenv, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4985738

Tasks: 17796162

Signature: t1:4985738:1493736355:4c9358fbf536661dc6b1f11c546332cda54f21b2
2017-05-02 07:48:32 -07:00
Martijn Pieters
f208a06c58 Add perma-failing test to stop hgbuild releases
Summary: The latest release of remotefilelog is failing to parse datapack files in the wild; until a fix can be deployed or a better test can be devised, this failing test will ensure we are not pushing out a broken release.

Test Plan: Check that rt test-remotefilelog-t17796162 fails

Reviewers: #sourcecontrol, mitrandir

Reviewed By: mitrandir

Subscribers: mitrandir, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4985575

Tasks: 17796162

Signature: t1:4985575:1493731275:19e7db182559e96d5310ec3e4ac221597553b97c
2017-05-02 06:21:44 -07:00
Stanislau Hlebik
9a02f3a792 infinitepush: use infinitepush path instead of default push
Summary:
As @quark suggested, it's better to use `infinitepush` path instead of
default-push, because `default-push` is facebook-specific configuration.

Test Plan: arc unit

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: quark, rmcelroy, mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4962324

Tasks: 15389402

Signature: t1:4962324:1493727112:35c45c57a527d2de5a35ea83e1031dc1908ac28e
2017-05-02 05:50:10 -07:00
David Soria Parra
aafcc1cfe6 p4fastimport: add retries to certain commands
Summary:
Add a simple contextmanager that allows retrying certain commands.
Sometimes our connection to Perforce aborts and we must restart the current
try. We have to be very careful about situations where we `yield` to not
accidentally yield the same value twice. For the moment we just enable this for
a very few commands that are prone to this (are called during long standing
import operations such as Filelog imports and changelist imports)

Test Plan:
1. test import of ovrsource.
2. rt test-p4* test-check*

Reviewers: #sourcecontrol, #idi, wlis

Reviewed By: wlis

Subscribers: wlis, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4980728

Signature: t1:4980728:1493675911:313b0f74983085688670e9d434968965f6e0ae8a
2017-05-01 20:43:11 -07:00
David Soria Parra
b70ec1d07e p4fastimport: make revs a set
Summary:
Make revs a set, just to clarify the semantics that a rev must not
show up twice.

Test Plan: rt test-p4* test-check*

Reviewers: #sourcecontrol, #idi, wlis

Reviewed By: wlis

Subscribers: wlis, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4980705

Signature: t1:4980705:1493675491:1633ff632410fedf4d535496b928e8ac85943ce2
2017-05-01 20:43:11 -07:00
David Soria Parra
aecf0fd2c7 p4fastimport: better progress description
Summary: Just a better description

Test Plan: rt test-p4* test-check*

Reviewers: #sourcecontrol, #idi, wlis

Reviewed By: wlis

Subscribers: wlis, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4980691

Signature: t1:4980691:1493675446:ebab713a32aa91265f1803ba54676bf075eb4b4e
2017-05-01 20:43:11 -07:00
David Soria Parra
646de74a72 p4fastimport: use --client in fstat call
Summary: Provide the --client option to pretend the client belongs to us.

Test Plan: rt test-p4* test-check*

Reviewers: #sourcecontrol, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4980666

Signature: t1:4980666:1493689890:f396595f9caa03f9f96b317e7df61589dfe29082
2017-05-01 20:43:11 -07:00
David Soria Parra
c428bfdaa8 p4fastimport: reuse the manifestnode we generated in previous iteration
Summary:
We are generating multiple manifests in memory. When we are trying to
read the previous manifest and doing a changelog lookup, we might trigger a bug
in Mercurial. Mercurial tries to use the 00changelog.d file instead of the .i file once it
reached a certain size. As this is all happening in one big transaction,
Mercruial is supposed to read this from in-memory, but this is broken atm. We
can circumvent this bug by reusing the manifestnode that we generated in the
previous iteration. This is more effective anyway.

Test Plan:
1. test import of ovrsource
2. rt test-p4* test-check*

Reviewers: durham, quark

Reviewed By: quark

Subscribers: quark, #idi, mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4980601

Signature: t1:4980601:1493689852:889a661c6fe606119247c8261dc567e7d361dacb
2017-05-01 20:43:11 -07:00
David Soria Parra
f514db7c65 p4fastimport: support action archive
Summary:
Support the action archive. 'archive' means that a revision was
"archived" to a different depot. We must ensure we support the action correctly
in order to have a smooth import.

Test Plan: run it && rt test-p4* test-check*

Reviewers: #sourcecontrol, #idi, wlis

Reviewed By: wlis

Subscribers: wlis, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4980571

Signature: t1:4980571:1493676115:ad0d3748c52747aeb6427fd25ece9d4987886936
2017-05-01 20:43:11 -07:00
David Soria Parra
305a284a00 p4fastimport: generate filelogs using fstat concurrently
Summary:
Generating case-correct filelogs using fstat leads to O(changelists)
calls to Perforce (and overall complexity of O(changelists*number of files),
which is slow. We want to run this using workers.

Test Plan: rt test-p4* test-check*

Reviewers: #sourcecontrol, quark

Reviewed By: quark

Subscribers: quark, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4963767

Signature: t1:4963767:1493349047:3eaddf6a3bb2ee06decaac48980c69b8645ebbed
2017-05-01 20:43:11 -07:00
David Soria Parra
bd16b38823 p4fastimport: add --limit option to process N changelists per transaction
Summary:
When passing --limit we are processing N Perforce changelists at a time.
The goal is to provide savepoints for large imports.

Test Plan: rt test-p4* test-check*

Reviewers: #sourcecontrol, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4980482

Signature: t1:4980482:1493688071:800a0bafda33a17cb2ef54c9f399db7055a8cbf9
2017-05-01 20:43:11 -07:00
David Soria Parra
fc6dc3be52 p4fastimport: adding debug message to filelog loading
Summary: Add debug output while loading filelogs when --debug is passed.

Test Plan: rt test-check* test-p4*

Reviewers: #sourcecontrol, #idi, ikostia

Reviewed By: ikostia

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4963656

Signature: t1:4963656:1493315731:f2b28dd06e4611d74e039d8b7805f72eb1bc16e4
2017-05-01 20:43:11 -07:00
Jun Wu
4b467776ad pushrebase: accept arbitary keywords for _peerorrepo
Summary:
stateful chg will add a new parameter to repo initialization. This diff makes
sure pushrebase works with that change.

Test Plan: codereview

Reviewers: #mercurial, durham

Reviewed By: durham

Subscribers: durham, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4978687

Signature: t1:4978687:1493689190:7119205d76a8ac26dbf2bcf0be9c74ced9fc16b4
2017-05-01 19:11:08 -07:00
Jun Wu
491a1ce44c datapack: fix a freememory issue
Summary:
The issue was introduced by the getmeta refactoring. It didn't get caught by
tests because tests didn't trigger freememory.

Test Plan: The fix was verified working on fbsource

Reviewers: #mercurial, durham

Reviewed By: durham

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4978835

Signature: t1:4978835:1493689136:296d425cf5d08b807b898c0e8cd881c9207c6359
2017-05-01 19:09:06 -07:00
Jun Wu
4a613bcaef datapack: update the format about metadata
Summary:
This diff makes 2 changes to v1 packfile metadata:

1. Move `key` in a metadata entry to before `size`.

```
old: [entry-size: 2 byte] [key: 1 byte] [data: var length]
new: [key: 1 byte] [data-size: 2 byte] [data: var length]
```

Previously `entry-size == 0` does not make sense.

2. Use binary to represent sizes, instead of ASCII.

Related utility methods are cleaned up a bit so it's harder to make mistakes.

Test Plan: Updated existing tests

Reviewers: #mercurial, durham

Reviewed By: durham

Subscribers: durham, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4983189

Signature: t1:4983189:1493689852:22d544d73ed63fac83f849786de035af304161ce
2017-05-01 19:03:25 -07:00
Jun Wu
00931e879b cdatapack: fix getdeltachain
Summary:
D4960035 used `GET_DELTA_CHAIN_CORRUPT` at a place where it should be
`GET_DELTA_CHAIN_LINK_CORRUPT`. The error was not caught by gcc but clang.

Test Plan: Build on OS X

Reviewers: durham

Reviewed By: durham

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4981954

Signature: t1:4981954:1493678436:3cb4779f14e64951c4d489ca30766888e7b0386a
2017-05-01 15:44:36 -07:00
Jun Wu
c4787f6acd perftest: add benchmark for datapack
Summary:
This shows how much overhead the `getmeta` call introduces.

Initial investigation shows the C version is about 10x as fast as than `get` or `getdeltachain`.

Test Plan: Run `hg perfdatapack` in an repo.

Reviewers: #mercurial, durham

Reviewed By: durham

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4909157

Signature: t1:4909157:1492667211:e82a14e12d04f60c0cee4124e83845ec166a78cc
2017-05-01 13:38:39 -07:00
Jun Wu
9a8545ebc1 cdatapack: implement getmeta for C module
Summary:
This diff implements getmeta in C and enables related tests.

Now all content stores support `getmeta`.

Test Plan: Run existing tests.

Reviewers: #mercurial, durham

Reviewed By: durham

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4960926

Signature: t1:4960926:1493611048:55095c32927fac74e698f21d47173cb8a7523fb6
2017-05-01 13:29:19 -07:00
Jun Wu
2ad4e3a191 cdatapack: avoid lz4decompress in getdeltachainlink
Summary:
This patch moves lz4decompress logic to a separate function from
getdeltachainlink.

This should solve a memory leak issue and speed up datapack entry iteration.
Practically, this means repack will be faster and take less memory.

Test Plan: `make clean local`, run `test-cstore.t`, and `test-remotefilelog-repack-fast.t`

Reviewers: #mercurial, durham

Reviewed By: durham

Subscribers: mjpieters, terrelln

Differential Revision: https://phabricator.intern.facebook.com/D4960035

Signature: t1:4960035:1493609520:a3c74bae92b8fff85ccadd9dd412a0c2b05573ac
2017-05-01 13:03:38 -07:00
Durham Goode
d602cf1e82 fixtests: update tests to match upstream
Upstream mutable-history has changed the output a bit.
2017-05-01 09:22:37 -07:00
Durham Goode
3ca5e9b375 treemanifest: put bundle sending behind a config flag
The bundle2 part was always being generated, even for non-treemanifest repos
(because fastmanifest imports the treemanifest module, so the b2partsgenerator
gets registered). This cause pushes to break in any repo with fastmanifest
enabled.
2017-04-30 22:05:09 -07:00
Jun Wu
e623b8f80c setup: be permissive about cython version
Summary:
It's reported Cython 0.20.1post0 version shipped by Ubuntu works.
However, I cannot build linelog using Cython at git commit 0e6e38ec8,
which is supposed to be the version used by that package.

That said, let's remove the explicit version check but run a real build to
detect whether Cython works or not.

[1]: https://bitbucket.org/facebook/hg-experimental/pull-requests/21/build-only-requiring-cython-020-not-022

Test Plan: Run `make local`.

Reviewers: #sourcecontrol, durham

Reviewed By: durham

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4972024

Signature: t1:4972024:1493407514:c5e0d21f92e866955b40600d1a1144f780e90c6e
2017-04-28 12:26:03 -07:00
Stanislau Hlebik
9f975aa8ec phabstatus: fix test because of arcanist configuration changes
Summary: We've made changes to arcanist configuration recently. Need to do the same in the tests.

Test Plan: arc unit

Reviewers: #scmquery, rmcelroy, ikostia

Reviewed By: ikostia

Subscribers: mjpieters, #sourcecontrol

Differential Revision: https://phabricator.intern.facebook.com/D4969670

Signature: t1:4969670:1493376937:536470162c52c8be953c813c1fea54f3e506f1e9
2017-04-28 03:58:08 -07:00
Mark Edson
36d60f5259 phabricator: make conduit auth more flexible
Summary:
We've modified arcanist a while ago so that if it doesn't find user/cert information for the specific URL that it's using, it just picks the user/cert information from any other URL in ~/.arcrc.  This is because it's essentially always the same user, and always the same cert, so there's really no point in being too picky.
This updates hg extension to be almost as careless.  It will attempt to find the matching user/cert, but if it doesn't work, it'll just pick any cert if available.

Test Plan:
Without this change, "hg ssl" in a recent version of www reports an warning because www/.arcconfig has phabricator.intern.facebook.com, where default ~/.arcrc's have phabricator.fb.com.
With this change, "hg ssl" succeeds to display revision information because it is once again able to authenticate with conduit

Reviewers: #phabricator, rmcelroy

Reviewed By: rmcelroy

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4964111

Tasks: 17683952

Signature: t1:4964111:1493323636:668b50ce2d20d720ba3de573de05be5251ce3310
2017-04-27 13:45:50 -07:00
Durham Goode
7e191da991 treemanifest: don't copy 00manifesttree.d during streaming clones
The old logic that prevented this clone only covered 00manifesttree.i. Let's
also cover 00manifesttree.d.
2017-04-27 12:03:56 -07:00
David Soria Parra
f8bb4e2122 lfs: add a 'null' store that doesn't require any setup
Summary:
We have to use the dummy store to ensure we don't try to open an HTTP
connection during push, which can fail and must not happen during p4fastimport.
However `dummy` requires some configuration. So just add an internal `null`
store that allows us to operate LFS in a mode that will ignore all large files
completely.

Test Plan: used it in p4fastimport

Reviewers: #sourcecontrol, quark

Reviewed By: quark

Subscribers: quark, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4963638

Signature: t1:4963638:1493316573:c012bd97794f9a57c3cf8c15d868a67ae3c03c31
2017-04-27 11:13:54 -07:00
Durham Goode
2b12f514c7 treemanifest: include tree pack during pushes
Summary:
To enable pushing between peers (and eventually pushing to the server), let's
teach bundle creation to include the trees being pushed.

Test Plan: Adds a test

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: quark, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4957456

Signature: t1:4957456:1493266296:67f98a2b3d691644bde9098a713d05266f349cde
2017-04-27 10:44:34 -07:00
Durham Goode
afacc9433b treemanifest: unify client and server tree history access patterns
Summary:
Previously the server would access the tree data in an adhoc manner. Sometimes
it would talk straight to revlogs, sometimes it would create stores and talk to
data packs. This patch makes it access trees the same way clients do, through
repo.svfs.manifestdatastore and manifesthistorystore.

This also cleans up the client store creation just a little and adds a
unionmetadatastore for unified history access.

Test Plan: Ran the tests

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4957441

Signature: t1:4957441:1493263349:e76d177f7a9f45343e6f984d6c0ae2c7cacba035
2017-04-27 10:44:34 -07:00