Summary:
The error was renamed to `InvalidPointer`. This diff updates error handling
to catch that instead.
Test Plan: arc unit
Reviewers: davidsp, #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5067496
Signature: t1:5067496:1494955510:5f0162a8cc6f1d0d83e3ab6319ec3202028684d8
Summary:
This simplifies code a bit, and could be useful for 3rd party code. We already
have `isbinary`, `islink`, `isexec`, `isabsent`. So another `is` method looks
fine.
Test Plan: Run existing tests
Reviewers: davidsp, #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5067369
Signature: t1:5067369:1494955466:ae17e310e743c704cc0d9bb73e0d2e82adcffff8
Summary:
The `filectx.cmp` fast path allows us to show "binary file changed" diff output
when two binaries have the same size but different content.
Test Plan: Added a test case
Reviewers: davidsp, #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5067266
Signature: t1:5067266:1494954783:a41e8213782afe24618ef2930c9576f21610fd3e
Summary:
When the server performs a repack, it would read all the data from all the
revlogs. This was very slow and expensive. Let's add a --incremental option that
makes it only read the revlog entries that have a linkrev newer than the latest
linkrev data that's already in a pack file.
Test Plan: Adds a test
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4997260
Signature: t1:4997260:1493904216:c4f5b6c9652bbf8f66c1bc2b2547c898324d22cd
Summary:
Calculating what blobs to upload should be fast since we use changelog `files`
information. Users will see `lfs: uploading ...` very soon. So let's downgrade
`lfs: computing set of blobs to upload` to a debug message to make `-v`
cleaner.
Test Plan: Updated existing test.
Reviewers: davidsp, #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5066600
Signature: t1:5066600:1494954115:c00925a1930f9b53e914078d32b6c4e4161099ee
Summary:
Downloading or uploading a single LFS object could take long. Instead of
showing a summary of what objects are downloaded or uploaded, print a message
per object if --verbose is provided.
Test Plan: Added a test
Reviewers: davidsp, #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5066327
Signature: t1:5066327:1494953894:3593436ff53eaff117a2932476dfc4b867fb9f17
Summary:
Previously, when an object is not found server-side, we error out during the
basic transfer API. This diffs move the check to the batch metadata API so the
check is preformed earlier.
That also means, when batch fetching many LFS objects, we could report multiple
objects being missing instead of just one.
Test Plan: Modified an existing test
Reviewers: davidsp, #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5065623
Signature: t1:5065623:1494952131:cb1df73e1cea21e07bfd0ec630bacec60e5b385c
Summary:
We are adding a simple transaction that works only with filelogs but
allows for concurrent access from multiple workers. This allows for a proper
rollback in case of a failure in a worker process, which previously would result
in bad data in the repositoriy.
Test Plan: rt test-p4* test-check*
Reviewers: #mercurial, durham, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5070340
Signature: t1:5070340:1494958313:b10b1eac5b42b36d1a587c4ae1c95fc2f8b5ad35
Summary:
The batch API allows server to return errors per object, handling them early
seems to be a good idea.
This is an attempt to distinguish "cannot upload - no permission" from
"no need to upload - already exist".
Unfortunately, `lfs-test-server` reference implementation does not provide
the error message so it's not reflected in tests.
Reviewers: #mercurial, davidsp, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5057816
Signature: t1:5057816:1494951879:07f2b67408e9d77255104b932baae2e50af9890b
Summary:
The LFS uploading or downloading process consists of 2 parts: a batch API to
get metadata about objects, followed by `len(objects)` requests.
This diff moves the first metadata API out from a giant `_batch` method to
make the code structure easier to maintain. Later diffs will further split
`_batch` method.
A side effect is `lfs: mapping blobs to #{action} URLs` message is removed.
More user-friendly message will be added back in a later patch.
Reviewers: #mercurial, davidsp, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5057541
Signature: t1:5057541:1494949989:d510c3d9369744e7ee776c0d0a26d263dfaf8352
Summary:
Displaying total bytes to upload is currently inaccurate - the server could
already have some objects so only a subset of selected objects will be
uploaded.
Besides, we pass `pointers` to related upload and download APIs so `total`
could be calculated from them.
This diff removes the inaccurate "need to upload" message and unnecessary
parameters. An accurate message will be added in a later patch.
Reviewers: #mercurial, davidsp, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5057470
Signature: t1:5057470:1494949554:07e637a8b58c894598b3f0c2dd53f80a5ade25a8
Summary:
Upstream shelve now writes simplekeyvaluefile for shelvedstate. This means
that tests which relied on ordered state files need to be adjusted.
Test Plan: - rt
Reviewers: durham, mitrandir, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D5069982
Summary: Added -U/--unified NUM to fbshow.py, and used that to set diff.unified=NUM.
Test Plan: hg show --help; hg show -U 1; hg show -U 2; hg show -U by induction.
Reviewers: mburman, simonfar, ikostia
Differential Revision: https://phabricator.intern.facebook.com/D5052156
Tasks: 10006036
Tags: python
Summary:
A try block is only effective for the HTTP error. Let's narrow it down to
just the HTTP request.
Test Plan: arc unit
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5057107
Signature: t1:5057107:1494634308:584877babc6a4705fa0740788dcf58affdf51bec
Summary:
There are `1+n` HTTP requests. `n` of them is handled. But the first one
about JSON metadata is not. Let's handle it.
Test Plan: arc unit
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5057053
Signature: t1:5057053:1494634184:fa0e562dc7879379106bf37915a2e5ea3cc01a4d
Summary:
The server or the HTTP library may provide more detailed error message.
Let's use them. `RequestFailedError` is also renamed to `LfsRemoteError` to
make it clear it's related to LFS and network.
Test Plan: arc unit
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5057014
Signature: t1:5057014:1494634052:00c1cb1b337a0e4cb92828cd49e95b7b0ff7e540
Summary:
It's caught, and re-raise with RequestFailedError. We can just raise
RequestFailedError directly.
Test Plan: arc unit
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5056886
Signature: t1:5056886:1494633307:3fdf2fab13e783224d343266a1278c41ed4394e8
Summary:
_batch is called by programmers. If action is not expected, it's a
ProgrammingError.
Test Plan: arc unit
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5056822
Signature: t1:5056822:1494634121:59737eb0d3f42d706ee8b0dc18ec7a4b49ada34f
Summary:
move p4fastimport under hgext3rd as that's where it belongs.
Also has the benefit that we package it up.
Test Plan: rt test-p4*
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5056664
Signature: t1:5056664:1494631758:4892f47922d8fbbd2c1f7793f3f29ee73fa8fa42
Summary:
Add a debugscanlfs command to p4fastimport. We want a similar command in
lfs but in this case we also want to do the reverse lookup of a
clientspec, etc.
Test Plan: $ unbuffer hg --config 'p4fastimport.useworker=False' debugscanlfs --debug --trace --client 'dsp-test' -A
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5055890
Signature: t1:5055890:1494629600:a20abf217abe51f26b179e0ebfdaf9230e3ad34d
Summary:
For long I have been wondering - we have `oid` already and an `oid` could
identify an object already, why do we need a `StoreID` which contains both
`oid` and `size`? It does not seem necessary?
I figured out the answer - the batch download or upload API *requires* size
information per object [1].
Since we will have LFS pointers during upload or download, we can just use
pointers. This diff removes `StoreID`, changes remote blob store to take
`pointers` directly, and local blob store to take `oid` directly.
As a side effect, `debuglfsupload` no longer takes `-o`, which seems fine
because `-r` should be more useful.
[1] https://github.com/git-lfs/git-lfs/blob/master/docs/api/batch.md
Test Plan: `arc unit`
Reviewers: davidsp, #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy
Differential Revision: https://phabricator.intern.facebook.com/D5043694
Signature: t1:5043694:1494496522:ea10f3dd12fc2ae3ab77b8f623eeead84dcc25fa
Summary:
Read the Git-LFS specification [1] and implement most checks to prevent
programming error and detect data corruption.
The new code should be stronger than what was before the refactoring, since the
old code only checks keys but not values.
[1]: https://github.com/git-lfs/git-lfs/blob/master/docs/spec.md
Test Plan: Added a new test
Reviewers: #mercurial, davidsp, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5046740
Signature: t1:5046740:1494543072:2f0dbfc07cd52a10572550a277d850bdf3f78d27
Summary:
Since we decided to only support GitHub's Git LFS specification, there is no
need to support multiple pointer types, so the code could be simplified.
The old code special case keys like `version`, `oid`, `hashalgo`, which makes
it longer than necessary. The new code is a rewrite treating everything as a
normal dict entry so the pointer class is much shorter: 76 -> 21 lines.
Data validation is temporary lost, which will be added back (and stronger) by
the next diff. It is separated to make review easier.
Test Plan: Run existing tests
Reviewers: #mercurial, davidsp, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy
Differential Revision: https://phabricator.intern.facebook.com/D5043547
Signature: t1:5043547:1494543031:ac1100939a10a79dfd749ab6ac9c3bb7fcd84dbf
Summary:
This diff adds a shared `p4setup.sh` that de-duplicates common logic among
tests. It also uses absolute path to make sure the extension being tested is
the version being developed.
The LFS test is also workarounded temporarily waiting for upstream change.
Test Plan: Run existing tests
Reviewers: #mercurial, davidsp
Reviewed By: davidsp
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5049279
Signature: t1:5049279:1494547832:28222fd2034115faca73860d6dd2f19206701aaa
Summary:
`rlog` is optional. This diff removes the following output from test output:
```
+ sh: rlog: command not found
```
Test Plan: Run existing test.
Reviewers: #mercurial, davidsp
Reviewed By: davidsp
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5047296
Signature: t1:5047296:1494535382:c57e554b57f28428a9d677d6e1089c81585e1bf2
Facebook probably doesn't want external users reaching out to them directly
for support, so this should be customizable to specify other possible courses
of action (especially in cases where githelp is further extended to document
git-related company-internal commands in other companies).
It looks like there is some flakeyness in racing pushes during this test. Seems
to be caused by hg serve loading the initial repo state while other processes
are writing new commits, which results in a missing master. Introducing a slight
sleep to the beginning of the transaction allows all the processes to load the
correct initial state and gets rid of the flakeyness.
Summary:
This hooks into the strip code to allow stripping revisions from treemanifests
as well. This will make it easier to clean up server repos when mistakes happen.
Test Plan: Adds a test
Reviewers: #mercurial
Differential Revision: https://phabricator.intern.facebook.com/D5031489
Summary:
Previously, when creating a new manifestrevlog for the trees, we attempted to
copy the opener options and insert the treemanifest value for the manifestrevlog
class to pick up. It turns out that the opener we copied was actually just a
wrapper around the same underlying object that the real repo used, so when we
did opener.options = oldopener.options.copy(), we were actually putting the new
copy on both the old and new opener. This meant we enabled treemanifest for the
entire repo.
This wasn't a problem in most cases because the manifestlog had already been
loaded. But in a pushrebase world, all the data structures are refreshed when
the lock is obtained (after waiting for the other push to finish). In this
situation, the refreshed repo.manifestlog was now a treemanifest so the second
commit was pushed to the main manifest as a tree.
The fix is to not use the opener.options hack to configure the tree. Instead we
are refactoring upstream to take an explicit options to the manifestrevlog
constructor. This patch makes our internal treemanifest extension use this new
api and adds a test for push contention.
Test Plan: Added a test. It failed before, and passes after.
Reviewers: #mercurial
Differential Revision: https://phabricator.intern.facebook.com/D5026019
Summary:
`filelog.filenode` could be `None` when it's unknown - like working copy. That
breaks `isbinary` test. Let's modify `_islfs` check to return False to disable
lfs code path in that case.
Test Plan: Added a new test.
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5034058
Signature: t1:5034058:1494431790:a02b69addf5d2f9b6b9a27ad71ed4f136b2bfd2b
Summary: `revdiff` should use raw revisions. 5d11b5ed in core hg is a similar fix.
Test Plan: Added a test.
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5031916
Signature: t1:5031916:1494366439:80d496ed6f69a784d345510147eab6c479496f84
Summary:
77b66b03be changed how pushrebase and obsmarkers worked, which apparently causes
this test to change. Updating the test.
Test Plan: ran it
Reviewers: quark
Reviewed By: quark
Differential Revision: https://phabricator.intern.facebook.com/D5030830
Signature: t1:5030830:1494360593:e0c9ce9a8466cb725b3dcc22a9161c9f02b00f15
Summary:
The config option was designed to be used server-side to avoid accessing remote
LFS blob stores. However, we now have `lfs.url=null:` as a clean alternative,
which will explicitly raise if the server ever tries to download remote
content. So `bypass` is no longer needed.
`bypass` could also be used for displaying raw content. That is doable using
`debugdata`.
Test Plan: Modified existing tests
Reviewers: #mercurial, rmcelroy, davidsp
Reviewed By: davidsp
Subscribers: davidsp, rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5009725
Signature: t1:5009725:1494260384:332d7bd658359c004342efd01e4f13b6fc5499b1
Summary:
`lfs.blobstore` is a path used to store local blobs. Some of the blobs are not
uploaded yet so they couldn't be easily discarded. Although blobs downloaded
from the server could be removed if necessary.
It does not make much sense to make this a config option - the data should be
stored reliably. It's also dangerous to allow write to arbitrary paths under
`repo.vfs`.
This diff makes the local blob store fixed path at `.hg/store/lfs/objects`.
The choice is similar to `.git/lfs/objects`, but with `store`, shared repo
could share their lfs stores.
Test Plan: Modified existing test.
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5009664
Signature: t1:5009664:1493978440:c3f6351d0ea0cca2ce3caa9f7260c5d65bcc0e5f
Summary: This makes it possible to upload lfs blobs without going through push.
Test Plan: The feature will be used in the next change.
Reviewers: #mercurial, davidsp, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5009622
Signature: t1:5009622:1494230385:0df33ed12c96bc91f6252398ce5091057fe32a21
Summary:
This makes it possible to reuse part of them - like uploading blobs for
given revisions without going through prepush hook.
`pointer.tostoreids()` was changed to `pointer.tostoreid()` to simplify
things a bit.
Unnecessary remoterepo assignment was removed.
Test Plan: `arc unit`
Reviewers: #mercurial, davidsp, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5009560
Signature: t1:5009560:1494230285:6469a2701baa8cfa4511a08149a37fc429733343
Summary:
lfs-test-server is a reference implementation by GitHub [1]. Testing against it
will give us more confidence.
[1]: https://github.com/git-lfs/lfs-test-server
Test Plan: Added a test
Reviewers: #mercurial, davidsp
Reviewed By: davidsp
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5009346
Signature: t1:5009346:1494042196:86e8c2e0e5c4296b1afc72efa84a9c1b9030e6fc
Summary:
The `lfs-test-server` reference implementation [1] validates the hash, and
40-byte sha256 hexdigest will not pass that check.
[1]: https://github.com/git-lfs/lfs-test-server
Test Plan: A test will be added in the next diff.
Reviewers: #mercurial, davidsp
Reviewed By: davidsp
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5009319
Signature: t1:5009319:1494006078:32950c5490935d9786553fc07d01f1fc92aacf25
Summary:
The request JSON data used in the batch API should have `size` field as an
integer, not a string. This was discovered when testing against GitHub's
`lfs-test-server` implementation [1]. The latter was written in Golang and has
strong type requirement, it will treat `{"size": "42"}` as `{"size": 0}`.
[1]: https://github.com/git-lfs/lfs-test-server
Test Plan:
An integration test with `lfs-test-server` will be added once remaining issues
are resolved.
Reviewers: #mercurial, ikostia, davidsp
Reviewed By: ikostia, davidsp
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5009248
Signature: t1:5009248:1493977299:5daf8d32cd8c933be71a41afcc7ff832eb7edb5a
On Mac, this currently fails with
cstore/uniondatapackstore.cpp:62:78: error: implicit conversion loses integer
precision: 'data_offset_t' (aka 'unsigned long long') to 'size_t'
(aka 'unsigned long') [-Werror,-Wshorten-64-to-32]
...ConstantStringRef((const char*)fulltextLink->delta, fulltextLink->delta_sz);
~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~^~~~~~~~
Summary: This will allow `-M` to have complex queries like `REV^'.
Test Plan: Test `scmutil.revsingle` in debugshell.
Reviewers: #mercurial, ikostia, rmcelroy
Reviewed By: ikostia, rmcelroy
Subscribers: ikostia, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5008983
Signature: t1:5008983:1493977159:7e105be95330b498d25033565be8594789406cae
Summary:
This was reported on IRC. People building absorb got:
lz4.h: No such file or directory
Test Plan:
`make clean`, and try `setup.py build` with:
- no parameters
- `--component absorb`, make sure lz4 is not needed
- `--component remotefilelog`, make sure it builds
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5014663
Signature: t1:5014663:1494023360:31ededc49de2dda45dfea4f62d6759cb19d819ef
