Summary: Fix lint errors and make the test pass.
Test Plan: arc unit
Reviewers: #sourcecontrol, sid0
Reviewed By: sid0
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5142477
Signature: t1:5142477:1496173824:bf941d895d3259fc788c09d4fefca7dddd11a33b
Summary:
Vendored using GPL2 license from https://bitbucket.org/marmoute/mutable-history
changeset c9f1118b33d60f8faa4b89988a8155c544f5bb0d without modification.
Test Plan: N/A. The code was imported as is.
Reviewers: #sourcecontrol, sid0
Reviewed By: sid0
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5142063
Signature: t1:5142063:1496173803:d0a9fa50a8423a531432ac3cff378b50952684b1
Summary:
This adds the `skiprevs` parameter to all `filectx.annotate` methods
to match the upstream change.
For fastannotate, things is a bit more complex since it's a mix of two
algorithms. For now we just fallback to the slow path for correctness.
I'll think about adding back a fast path later.
Test Plan: arc unit
Reviewers: sid0, #mercurial, quark
Subscribers: stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5128392
Summary:
copytracing extension is going to be simpler than I expected in the beginning.
There is no need to separate folder, let's move it to hgext3rd
Test Plan: arc unit
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D5146585
Tasks: 18508761
Signature: t1:5146585:1496147733:ad6ad14f663a8b9bf4b687f0767395321f9840b7
Summary:
`revlog.ancestor` is expensive on long changelog, but is only called a few
(about 30) times for smartlog usage. Therefore we could simply cache the
result in a key-value database.
This speeds up smartlog by about 200ms.
Test Plan: Added a new test
Reviewers: #mercurial, stash
Reviewed By: stash
Subscribers: stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5135746
Signature: t1:5135746:1495783503:411260551fd29fda42c9fc809b56f8f77fb2eaf4
Summary:
Previously, the tracer extracts file name, function name from Python code
objects and stores them as std::string. And frame the de-duplication logic
hashes those strings, which is actually quite expensive.
`hg id` takes 1.0 seconds without frame de-duplication, and 2.5 seconds with
de-duplication.
This diff assumes Python code object is unique (i.e. Python won't generate two
code objects for a single code segment). That seems reasonable and `lsprof`
seems to make a same assumption.
With that assumption, just store `PyCodeObject` (contains file, method name and
line number) instead of `std::string` (for file, method names) and line numbers
for code identity and use the raw address of `PyCodeObject` to do frame
de-duplication.
That is helpful during profiling and de-duplication:
- During profiling, no need to copy strings (convert to std::string),
therefore less overhead
- During frame de-duplication, no need to read long strings and hash them,
therefore much faster
Debug flag during compilation is added to make debugging easier as the code
starts to mix of C++ and Python objects, which means mistakes could be made
more easily.
Test Plan:
With this patch, `hg id` with frame de-duplication takes 1.0 seconds.
There is no visible overhead doing de-duplication. Cheers!
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5112636
Signature: t1:5112636:1495562232:dd0724198ebb7f2eb4e9fd3a83517fc9160fa01e
Summary: This is what extensions should belong to.
Test Plan: Changed existing test.
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5113042
Signature: t1:5113042:1495561987:d1af0f7cebfdb8b0c1a6bb71cdc826d817a4e441
Summary:
Upstream has added an optional operation argument to createmarkers, which we
need to also accept. This argument is currently only processed if a config
option is set, so for now we will still put the operation in the metadata dict
manually.
Test Plan: Ran the tests
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5106511
Signature: t1:5106511:1495485044:23d45d9ba389cf0d49b31705ae36902310c5056c
Summary: Upstream has deprecated begin/endparentchange in favor of a context object.
Test Plan: Ran the tests
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5106500
Signature: t1:5106500:1495485058:8377335a0f678de7bb50cb93fff29bcfeb53fe1d
Summary: Upstream has deprecated cmdutil.commands() in favor of registrar.commands()
Test Plan: Ran the tests
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5106486
Signature: t1:5106486:1495485074:0e20f00622cc651e8c9dda837f84dd84cc51099e
Summary:
Previously, the progress bar is file-level - it moves when a file is transferred.
When uploading or downloading a single giant file, progress bar matters. And
this diff adds it to make people more patient.
Test Plan:
Manually upload and download several big files.
Make sure the progress bar appears in both cases.
Also make sure the output is sane with `-v`.
Reviewers: #mercurial, davidsp, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5100014
Signature: t1:5100014:1495316719:c500ccd63d3a495e41575cadd72419a41f5fb259
Summary:
This allows automation to know which modified or added files are LFS for
specific changesets.
Test Plan: Added a test
Reviewers: #mercurial, davidsp
Reviewed By: davidsp
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5091591
Signature: t1:5091591:1495163606:138638d8ccc57b8ed6c1e324750ec5dc15c198a0
Summary:
When remotestore is a null store, uploading is a no-op and can be skipped.
This diff makes it so.
Test Plan: arc unit
Reviewers: #mercurial, davidsp
Reviewed By: davidsp
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5090686
Signature: t1:5090686:1495144964:956549307912f4a44201aa305959bf8de26ac028
Summary:
conduit HTTP POST request did not have correct content-type header set.
Now they do. Ideally we should use python-request package to do HTTP
but I do not know enough how to import these dependencies so I just went
with the minimal changes.
Test Plan:
ran hg log from dev environment and eventually got it all working.
{P57411895}
Reviewers: simpkins
Reviewed By: simpkins
Subscribers: net-systems-diffs@fb.com, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5080725
Tasks: 18294945
Signature: t1:5080725:1495052201:1bbd6edad4c1c4b3482a7479259460a815947630
Summary:
`repair.strip` removes required revisions and their children unconditionally.
That means innocent children may be removed. This patch fixes it.
We may want to have an in-core function for this feature to not strip innocent
children.
Thanks Zibi Braniecki from Mozilla for reporting [1]!
[1]: https://bitbucket.org/facebook/hg-experimental/issues/6/hg-absorb-merges-diverged-commits
Test Plan: Added a test
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5073342
Signature: t1:5073342:1494972231:ca2a9881415119047771c8f35e059f4bfb1749e5
Summary:
When a bundle is created, the bundle could be exchanged via copy and possibly
eventually reach to a publishing repo. If we don't upload LFS blobs, hg server
could have revisions that can never be checked out or verified.
So let's just assume bundles generated by `hg bundle` will be public, and
upload LFS blobs automatically, without depending on other code review tooling.
Note: there is a `preoutgoing` hook which will be triggered in this case,
however it's not useful since it does not have the `outgoing` information.
Test Plan: Will add a new test
Reviewers: davidsp, #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5050945
Signature: t1:5050945:1494958648:0c46ab1d85755838ba189bbb1e0673882922bb58
Summary:
`util.py` only contains 2 small things: `lfsvfs` and `sha256`. Both of them
only have one user (in terms of files). Therefore just move the code to
related files to make it simpler.
Test Plan: arc unit
Reviewers: davidsp, #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5067862
Signature: t1:5067862:1494955824:b99021f1bb44568dc0c738b0e472eb732f2bc91a
Summary:
The error was renamed to `InvalidPointer`. This diff updates error handling
to catch that instead.
Test Plan: arc unit
Reviewers: davidsp, #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5067496
Signature: t1:5067496:1494955510:5f0162a8cc6f1d0d83e3ab6319ec3202028684d8
Summary:
This simplifies code a bit, and could be useful for 3rd party code. We already
have `isbinary`, `islink`, `isexec`, `isabsent`. So another `is` method looks
fine.
Test Plan: Run existing tests
Reviewers: davidsp, #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5067369
Signature: t1:5067369:1494955466:ae17e310e743c704cc0d9bb73e0d2e82adcffff8
Summary:
The `filectx.cmp` fast path allows us to show "binary file changed" diff output
when two binaries have the same size but different content.
Test Plan: Added a test case
Reviewers: davidsp, #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5067266
Signature: t1:5067266:1494954783:a41e8213782afe24618ef2930c9576f21610fd3e
Summary:
Calculating what blobs to upload should be fast since we use changelog `files`
information. Users will see `lfs: uploading ...` very soon. So let's downgrade
`lfs: computing set of blobs to upload` to a debug message to make `-v`
cleaner.
Test Plan: Updated existing test.
Reviewers: davidsp, #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5066600
Signature: t1:5066600:1494954115:c00925a1930f9b53e914078d32b6c4e4161099ee
Summary:
Downloading or uploading a single LFS object could take long. Instead of
showing a summary of what objects are downloaded or uploaded, print a message
per object if --verbose is provided.
Test Plan: Added a test
Reviewers: davidsp, #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5066327
Signature: t1:5066327:1494953894:3593436ff53eaff117a2932476dfc4b867fb9f17
Summary:
Previously, when an object is not found server-side, we error out during the
basic transfer API. This diffs move the check to the batch metadata API so the
check is preformed earlier.
That also means, when batch fetching many LFS objects, we could report multiple
objects being missing instead of just one.
Test Plan: Modified an existing test
Reviewers: davidsp, #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5065623
Signature: t1:5065623:1494952131:cb1df73e1cea21e07bfd0ec630bacec60e5b385c
Summary:
We are adding a simple transaction that works only with filelogs but
allows for concurrent access from multiple workers. This allows for a proper
rollback in case of a failure in a worker process, which previously would result
in bad data in the repositoriy.
Test Plan: rt test-p4* test-check*
Reviewers: #mercurial, durham, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5070340
Signature: t1:5070340:1494958313:b10b1eac5b42b36d1a587c4ae1c95fc2f8b5ad35
Summary:
The batch API allows server to return errors per object, handling them early
seems to be a good idea.
This is an attempt to distinguish "cannot upload - no permission" from
"no need to upload - already exist".
Unfortunately, `lfs-test-server` reference implementation does not provide
the error message so it's not reflected in tests.
Reviewers: #mercurial, davidsp, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5057816
Signature: t1:5057816:1494951879:07f2b67408e9d77255104b932baae2e50af9890b
Summary:
The LFS uploading or downloading process consists of 2 parts: a batch API to
get metadata about objects, followed by `len(objects)` requests.
This diff moves the first metadata API out from a giant `_batch` method to
make the code structure easier to maintain. Later diffs will further split
`_batch` method.
A side effect is `lfs: mapping blobs to #{action} URLs` message is removed.
More user-friendly message will be added back in a later patch.
Reviewers: #mercurial, davidsp, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5057541
Signature: t1:5057541:1494949989:d510c3d9369744e7ee776c0d0a26d263dfaf8352
Summary:
Displaying total bytes to upload is currently inaccurate - the server could
already have some objects so only a subset of selected objects will be
uploaded.
Besides, we pass `pointers` to related upload and download APIs so `total`
could be calculated from them.
This diff removes the inaccurate "need to upload" message and unnecessary
parameters. An accurate message will be added in a later patch.
Reviewers: #mercurial, davidsp, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5057470
Signature: t1:5057470:1494949554:07e637a8b58c894598b3f0c2dd53f80a5ade25a8
Summary: Added -U/--unified NUM to fbshow.py, and used that to set diff.unified=NUM.
Test Plan: hg show --help; hg show -U 1; hg show -U 2; hg show -U by induction.
Reviewers: mburman, simonfar, ikostia
Differential Revision: https://phabricator.intern.facebook.com/D5052156
Tasks: 10006036
Tags: python
Summary:
A try block is only effective for the HTTP error. Let's narrow it down to
just the HTTP request.
Test Plan: arc unit
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5057107
Signature: t1:5057107:1494634308:584877babc6a4705fa0740788dcf58affdf51bec
Summary:
There are `1+n` HTTP requests. `n` of them is handled. But the first one
about JSON metadata is not. Let's handle it.
Test Plan: arc unit
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5057053
Signature: t1:5057053:1494634184:fa0e562dc7879379106bf37915a2e5ea3cc01a4d
Summary:
The server or the HTTP library may provide more detailed error message.
Let's use them. `RequestFailedError` is also renamed to `LfsRemoteError` to
make it clear it's related to LFS and network.
Test Plan: arc unit
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5057014
Signature: t1:5057014:1494634052:00c1cb1b337a0e4cb92828cd49e95b7b0ff7e540
Summary:
It's caught, and re-raise with RequestFailedError. We can just raise
RequestFailedError directly.
Test Plan: arc unit
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5056886
Signature: t1:5056886:1494633307:3fdf2fab13e783224d343266a1278c41ed4394e8
Summary:
_batch is called by programmers. If action is not expected, it's a
ProgrammingError.
Test Plan: arc unit
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5056822
Signature: t1:5056822:1494634121:59737eb0d3f42d706ee8b0dc18ec7a4b49ada34f
Summary:
move p4fastimport under hgext3rd as that's where it belongs.
Also has the benefit that we package it up.
Test Plan: rt test-p4*
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5056664
Signature: t1:5056664:1494631758:4892f47922d8fbbd2c1f7793f3f29ee73fa8fa42
Summary:
For long I have been wondering - we have `oid` already and an `oid` could
identify an object already, why do we need a `StoreID` which contains both
`oid` and `size`? It does not seem necessary?
I figured out the answer - the batch download or upload API *requires* size
information per object [1].
Since we will have LFS pointers during upload or download, we can just use
pointers. This diff removes `StoreID`, changes remote blob store to take
`pointers` directly, and local blob store to take `oid` directly.
As a side effect, `debuglfsupload` no longer takes `-o`, which seems fine
because `-r` should be more useful.
[1] https://github.com/git-lfs/git-lfs/blob/master/docs/api/batch.md
Test Plan: `arc unit`
Reviewers: davidsp, #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy
Differential Revision: https://phabricator.intern.facebook.com/D5043694
Signature: t1:5043694:1494496522:ea10f3dd12fc2ae3ab77b8f623eeead84dcc25fa
