Summary:
As @quark suggested, it's better to use `infinitepush` path instead of
default-push, because `default-push` is facebook-specific configuration.
Test Plan: arc unit
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: quark, rmcelroy, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4962324
Tasks: 15389402
Signature: t1:4962324:1493727112:35c45c57a527d2de5a35ea83e1031dc1908ac28e
Summary:
Add a simple contextmanager that allows retrying certain commands.
Sometimes our connection to Perforce aborts and we must restart the current
try. We have to be very careful about situations where we `yield` to not
accidentally yield the same value twice. For the moment we just enable this for
a very few commands that are prone to this (are called during long standing
import operations such as Filelog imports and changelist imports)
Test Plan:
1. test import of ovrsource.
2. rt test-p4* test-check*
Reviewers: #sourcecontrol, #idi, wlis
Reviewed By: wlis
Subscribers: wlis, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4980728
Signature: t1:4980728:1493675911:313b0f74983085688670e9d434968965f6e0ae8a
Summary:
Make revs a set, just to clarify the semantics that a rev must not
show up twice.
Test Plan: rt test-p4* test-check*
Reviewers: #sourcecontrol, #idi, wlis
Reviewed By: wlis
Subscribers: wlis, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4980705
Signature: t1:4980705:1493675491:1633ff632410fedf4d535496b928e8ac85943ce2
Summary: Provide the --client option to pretend the client belongs to us.
Test Plan: rt test-p4* test-check*
Reviewers: #sourcecontrol, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4980666
Signature: t1:4980666:1493689890:f396595f9caa03f9f96b317e7df61589dfe29082
Summary:
We are generating multiple manifests in memory. When we are trying to
read the previous manifest and doing a changelog lookup, we might trigger a bug
in Mercurial. Mercurial tries to use the 00changelog.d file instead of the .i file once it
reached a certain size. As this is all happening in one big transaction,
Mercruial is supposed to read this from in-memory, but this is broken atm. We
can circumvent this bug by reusing the manifestnode that we generated in the
previous iteration. This is more effective anyway.
Test Plan:
1. test import of ovrsource
2. rt test-p4* test-check*
Reviewers: durham, quark
Reviewed By: quark
Subscribers: quark, #idi, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4980601
Signature: t1:4980601:1493689852:889a661c6fe606119247c8261dc567e7d361dacb
Summary:
Support the action archive. 'archive' means that a revision was
"archived" to a different depot. We must ensure we support the action correctly
in order to have a smooth import.
Test Plan: run it && rt test-p4* test-check*
Reviewers: #sourcecontrol, #idi, wlis
Reviewed By: wlis
Subscribers: wlis, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4980571
Signature: t1:4980571:1493676115:ad0d3748c52747aeb6427fd25ece9d4987886936
Summary:
Generating case-correct filelogs using fstat leads to O(changelists)
calls to Perforce (and overall complexity of O(changelists*number of files),
which is slow. We want to run this using workers.
Test Plan: rt test-p4* test-check*
Reviewers: #sourcecontrol, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4963767
Signature: t1:4963767:1493349047:3eaddf6a3bb2ee06decaac48980c69b8645ebbed
Summary:
When passing --limit we are processing N Perforce changelists at a time.
The goal is to provide savepoints for large imports.
Test Plan: rt test-p4* test-check*
Reviewers: #sourcecontrol, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4980482
Signature: t1:4980482:1493688071:800a0bafda33a17cb2ef54c9f399db7055a8cbf9
Summary:
stateful chg will add a new parameter to repo initialization. This diff makes
sure pushrebase works with that change.
Test Plan: codereview
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4978687
Signature: t1:4978687:1493689190:7119205d76a8ac26dbf2bcf0be9c74ced9fc16b4
Summary:
The issue was introduced by the getmeta refactoring. It didn't get caught by
tests because tests didn't trigger freememory.
Test Plan: The fix was verified working on fbsource
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4978835
Signature: t1:4978835:1493689136:296d425cf5d08b807b898c0e8cd881c9207c6359
Summary:
This diff makes 2 changes to v1 packfile metadata:
1. Move `key` in a metadata entry to before `size`.
```
old: [entry-size: 2 byte] [key: 1 byte] [data: var length]
new: [key: 1 byte] [data-size: 2 byte] [data: var length]
```
Previously `entry-size == 0` does not make sense.
2. Use binary to represent sizes, instead of ASCII.
Related utility methods are cleaned up a bit so it's harder to make mistakes.
Test Plan: Updated existing tests
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4983189
Signature: t1:4983189:1493689852:22d544d73ed63fac83f849786de035af304161ce
Summary:
D4960035 used `GET_DELTA_CHAIN_CORRUPT` at a place where it should be
`GET_DELTA_CHAIN_LINK_CORRUPT`. The error was not caught by gcc but clang.
Test Plan: Build on OS X
Reviewers: durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4981954
Signature: t1:4981954:1493678436:3cb4779f14e64951c4d489ca30766888e7b0386a
Summary:
This shows how much overhead the `getmeta` call introduces.
Initial investigation shows the C version is about 10x as fast as than `get` or `getdeltachain`.
Test Plan: Run `hg perfdatapack` in an repo.
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4909157
Signature: t1:4909157:1492667211:e82a14e12d04f60c0cee4124e83845ec166a78cc
Summary:
This diff implements getmeta in C and enables related tests.
Now all content stores support `getmeta`.
Test Plan: Run existing tests.
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4960926
Signature: t1:4960926:1493611048:55095c32927fac74e698f21d47173cb8a7523fb6
Summary:
This patch moves lz4decompress logic to a separate function from
getdeltachainlink.
This should solve a memory leak issue and speed up datapack entry iteration.
Practically, this means repack will be faster and take less memory.
Test Plan: `make clean local`, run `test-cstore.t`, and `test-remotefilelog-repack-fast.t`
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: mjpieters, terrelln
Differential Revision: https://phabricator.intern.facebook.com/D4960035
Signature: t1:4960035:1493609520:a3c74bae92b8fff85ccadd9dd412a0c2b05573ac
The bundle2 part was always being generated, even for non-treemanifest repos
(because fastmanifest imports the treemanifest module, so the b2partsgenerator
gets registered). This cause pushes to break in any repo with fastmanifest
enabled.
Summary:
It's reported Cython 0.20.1post0 version shipped by Ubuntu works.
However, I cannot build linelog using Cython at git commit 0e6e38ec8,
which is supposed to be the version used by that package.
That said, let's remove the explicit version check but run a real build to
detect whether Cython works or not.
[1]: https://bitbucket.org/facebook/hg-experimental/pull-requests/21/build-only-requiring-cython-020-not-022
Test Plan: Run `make local`.
Reviewers: #sourcecontrol, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4972024
Signature: t1:4972024:1493407514:c5e0d21f92e866955b40600d1a1144f780e90c6e
Summary: We've made changes to arcanist configuration recently. Need to do the same in the tests.
Test Plan: arc unit
Reviewers: #scmquery, rmcelroy, ikostia
Reviewed By: ikostia
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4969670
Signature: t1:4969670:1493376937:536470162c52c8be953c813c1fea54f3e506f1e9
Summary:
We've modified arcanist a while ago so that if it doesn't find user/cert information for the specific URL that it's using, it just picks the user/cert information from any other URL in ~/.arcrc. This is because it's essentially always the same user, and always the same cert, so there's really no point in being too picky.
This updates hg extension to be almost as careless. It will attempt to find the matching user/cert, but if it doesn't work, it'll just pick any cert if available.
Test Plan:
Without this change, "hg ssl" in a recent version of www reports an warning because www/.arcconfig has phabricator.intern.facebook.com, where default ~/.arcrc's have phabricator.fb.com.
With this change, "hg ssl" succeeds to display revision information because it is once again able to authenticate with conduit
Reviewers: #phabricator, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4964111
Tasks: 17683952
Signature: t1:4964111:1493323636:668b50ce2d20d720ba3de573de05be5251ce3310
Summary:
We have to use the dummy store to ensure we don't try to open an HTTP
connection during push, which can fail and must not happen during p4fastimport.
However `dummy` requires some configuration. So just add an internal `null`
store that allows us to operate LFS in a mode that will ignore all large files
completely.
Test Plan: used it in p4fastimport
Reviewers: #sourcecontrol, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4963638
Signature: t1:4963638:1493316573:c012bd97794f9a57c3cf8c15d868a67ae3c03c31
Summary:
To enable pushing between peers (and eventually pushing to the server), let's
teach bundle creation to include the trees being pushed.
Test Plan: Adds a test
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4957456
Signature: t1:4957456:1493266296:67f98a2b3d691644bde9098a713d05266f349cde
Summary:
Previously the server would access the tree data in an adhoc manner. Sometimes
it would talk straight to revlogs, sometimes it would create stores and talk to
data packs. This patch makes it access trees the same way clients do, through
repo.svfs.manifestdatastore and manifesthistorystore.
This also cleans up the client store creation just a little and adds a
unionmetadatastore for unified history access.
Test Plan: Ran the tests
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4957441
Signature: t1:4957441:1493263349:e76d177f7a9f45343e6f984d6c0ae2c7cacba035
Summary:
Previously, history stores only had getancestors() apis which returned all the
ancestors. This was expensive if there was a lot of ancestors, like for the root
tree of treemanifests. Let's add an api for accessing a single history entry.
This will be useful in pack generation for only fetching the history related to
the trees we're sending at that time.
Test Plan: Added a test
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4957432
Signature: t1:4957432:1493263124:a155ac5a70c35f7e25a5cc48c9d9c2126d4c5858
Summary:
Previously, a local commit would only write data packs, and it just threw away
the history data entirely. Let's add history packs and record them on commit.
Test Plan:
The tests are updated to show these new packs. In some cases the
datapacks got smaller as well, since they can now take advantage of history data
for delta choices.
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4956105
Signature: t1:4956105:1493265399:d3fa1052c207fba0045cbb92b4d833d18d48e099
Summary:
Previously, the logic that added data to a mutable history pack was required to
add it in the correct order (all entries for a certain file at once, and in
newest-first order). This required the callers to jump through weird hoops if
the data came in out of order or at different times in the transaction.
This patch moves the ordering logic to be inside MutableHistoryPack, so callers
can add the data in any order they wish, and it will get sorted before being
serialized.
This does add memory pressure to things that read a lot of history, like repack.
If this becomes a problem we may want to add a 'historypack.flush()' api that
let's us tell the history pack it's ok to flush it's current contents to disk.
Test Plan: Ran the tests
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4956096
Signature: t1:4956096:1493264693:a2275a49e35565d4b11244e3e5dd82c25de7e16e
Summary:
It's possible for history packs to not have all of history, so let's allow that
during repacks.
Also fix one exception to use hex()
Test Plan: A future patch has tests that exposed this
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4956078
Signature: t1:4956078:1493264261:81c12520b7352d5040cdb027509c975678c10069
Summary:
The code that creates new trees when the user commits only creates new trees if
the parent commit has a tree already. If the commit is the very first commit in
the repo (i.e. p1 == nullid), then the parent tree is empty. The `if tree:`
check considered this not-a-tree and therefore the first commit did not generate
a tree. Let's switch to `if tree is None:` to handle this.
A future patch adds tests around peer-to-peer pushing/pulling that encounters
this.
Test Plan: Future patch tests caught this
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4956069
Signature: t1:4956069:1493263952:c3d59f3e48dda623fd8c67630cb4bcd474e63518
Summary:
The remote store needs to throw a KeyError if it can't find the node. Returning
None was just causing breakages up stream and meant the union store did not move
on to the next store. This was caught by peer-to-peer pushing tests in a future
patch.
Test Plan: Future tests caught this
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4956062
Signature: t1:4956062:1493264179:432a3add7c0501804520e9abc92e33ba8fec4384
Summary:
The old func api parsed the getpack arguments internally. Let's move that out to
the wire protocol level, so in future patches we can reuse generatepackstream
more easily.
Test Plan: Ran the tests
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4956038
Signature: t1:4956038:1493265461:e7ed1ad659fb0639c325df3f8352dbe06f650740
Summary:
Previously the wire pack protocol was a custom stream of bytes. This
patch moves it to be within bundle2 so we can take advantage of features it has
(versioning, arguments, mid-stream error messaging, including packs in other
bundles, etc). This will be useful in future patches where we want to allow
peer-to-peer exchange of client only tree manifest data.
This also removes all the custom argument encoding/decoding.
Test Plan:
Ran the tests. Order changed but the content remains the same and the
order is consistent.
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4924718
Signature: t1:4924718:1493050942:2734cfdbca0b0b0a3f3be1ee8e224f634553c305
Summary:
A future patch will move the pack wireprotocol to use bundle2. In this new world
we'll be given a file handle instead of a remote peer, so let's switch the
utility methods to work on a file handle instead.
Test Plan: Ran the tests
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4924553
Signature: t1:4924553:1493050882:7a9ee8b282bf47ef393362dd0114d801dc2a68d5
Summary:
Adds a config option that will automatically download the last N trees from the
server during pull. It efficiently decides what bases to set, based on what
commit parents it already has.
Test Plan: Added a test
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4923211
Signature: t1:4923211:1493200730:39872db6aabb48c12131ff8501a950c8ee60e2b0
Summary:
On Windows `arc install-certificate` writes the cert to APPDATA:
https://fburl.com/i7fpssf8, so that is where we need to look for it.
Test Plan: - make this change, see that `hg ssl` shows commit status locally
Reviewers: davidsp, andrasbelo, #sourcecontrol
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4962334
Tasks: 17511545
Summary:
This allows a third-party tool like Buck report more detail on what a sparse
command actually achieved.
Test Plan: Run the test suite.
Reviewers: #sourcecontrol, durham, rmcelroy, ikostia
Reviewed By: ikostia
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4962550
Tasks: 17150508
Tags: autosparse
Signature: t1:4962550:1493307403:d14e16d079121688a4eb69f0d69cd02ab55c4936
Summary:
The test covers common workflows like clone, commit, push, update, pull. It
exercises the remotefilelog plain store and Python datapack store to make sure
they won't lose the revlog flag. The test also tries to verify rename works
correctly.
Since the lfs extension may be eventually upstreamed, it seems a good idea to
make remotefilelog call `lfs.wrapfilelog` so lfs is free from remotefilelog
code.
Test Plan: Added a test
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4904281
Signature: t1:4904281:1492560308:5fd9f214ada6de795735ea7d737d30c1bf39812a
Summary:
Filelog methods like `addrevision`, `revision(raw=True)` are needed for flag
processor (lfs) to work correctly. Add them in remotefilelog so lfs wrapper
code could replace them.
Test Plan:
Run existing tests. Stronger tests and lfs integration test will be added when
this area is more complete.
Reviewers: #mercurial, durham, rmcelroy
Reviewed By: durham
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4903959
Signature: t1:4903959:1493163809:5ebd88fac21d8225a12ce68bfc63a2867ee43769
Summary:
This diffs add a `getmeta` method to all content stores. The cdatapack code is
modified to pass the tests, it needs further change to support `getmeta`.
The datapack format is bumped to v1 from v0. For v1, we append a `metadata`
dict at the end of each revision. The dict is currently used to store revlog
flags and rawsize of raw revlog fulltext. In the future we can put more data
like a second hash etc, without changing API or format again.
This diff focuses on correctness. A datapack caching layer to speed up
`getmeta` will be added later.
Tests are updated since we write new v1 packfile now and the format change
leads to different content and packfile names.
`Makefile`, `ls-l.py` are added to make tests easier to maintain.
Test Plan: Updated existing tests.
Reviewers: #mercurial, rmcelroy, durham
Reviewed By: durham
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4903917
Signature: t1:4903917:1493255844:7ef5d487096cd2f78f2aaae672a68d49f33632ee
Summary: Today is a bad day. I don't even know how this got in there.
Test Plan: y
Reviewers: durham, #mercurial
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4957784
Summary: Otherwise, the test requires hg-dev and will break third-party developers.
Test Plan: Reran test
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4957662
Signature: t1:4957662:1493242115:7e83b130791b07b1e45f04ce199e5c4ee9d58d77
Summary:
To my deep shame, I discovered today that `arc land` does not rerun `arc unit` if you introduce post-review changes.
Sorry!
Test Plan: Ran arc unit this time :)
Reviewers: quark, #mercurial, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4957528
Signature: t1:4957528:1493241560:9208e745ac92c0e771b0a4b4bac91d6c1feed025
Summary:
allows users to have JSON progress bar information written to a path
Controlled by the `ui.progressfile` config. Mercurial will overwrite this file
each time the progress bar is updated.
The schema of this file is (JSON):
- topics: array of topics from oldest to newest. (last is always the active one)
- state: map of topic names to objects with keys:
- topic (e.g. "changesets", "manifests")
- pos: which item number out of <total> we're processing
- total: total number of items
- unit: name of the type of unit being processed (e.g., "changeset")
- item: the active item being processed (e.g., "changeset #5")
- active: whether this is the currently active progress bar
- units_per_sec: if active, how many <unit>s per sec we're processing
- speed_str: if active, a human-readable string of how many <unit>s per sec
we're processing
- estimate_sec: an estimate of how much time is left, in seconds
- estimate_str: if active, a human-readable string estimate of how much time
is left (e.g. "2m30s")
Test Plan: added a test
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: rmcelroy, quark, stash, asriram, mjpieters, durham
Differential Revision: https://phabricator.intern.facebook.com/D4752788
Signature: t1:4752788:1493233878:b49095237d32233c78cd0e0aaaa7b94e0e9e7011
Summary:
To be able to bump version and change formats, the related constants need to
be moved to individual classes. So a class (ex. datapack) can be subclassed to
handle different formats.
Test Plan: `arc unit`
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4927284
Signature: t1:4927284:1493152641:e3274dd735d50baf193b7615dd314f4e6cf161f0
Summary:
Make the unpacked file format to include the revlog flag information, and make
the getfile(s) protocol support it.
Note: The `getpackv1` protocol and packfile format is not changed yet.
Test Plan:
Run existing tests. Stronger tests and lfs integration test will be added when
this area is more complete.
Reviewers: #mercurial, rmcelroy, durham
Reviewed By: durham
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4903772
Signature: t1:4903772:1493152451:ab393b0208f0eee199ffc4c8fcfdfd5dd6d0f3ac
Summary: second time's the charm
Test Plan: make local, wait for contbuild
Reviewers: #mercurial, mitrandir
Reviewed By: mitrandir
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4955570
Signature: t1:4955570:1493230082:3493ac29ff8d04a98e50f83febc9dd827e9b4a7c
