Summary:
This diff disables or rewrites problematic parts in linelog code on Windows -
`mmap`, `ftruncate`, `sysconf`, `htonl` so the linelog module could be built
without on-disk linelog support. That means `absorb` can work. `fastannotate`
needs extra effort.
Test Plan:
`python setup.py install` and check in Python shell:
```
>>> import linelog
>>> linelog.linelog()
<open linelog (in-memory) at 0x3ab07bb5c8>
```
Also check the test pass:
```
fb-hgext\linelog\pyext>%PYTHONHOME%\python test-random-edits.py
```
Reviewers: #mercurial, ikostia
Reviewed By: ikostia
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5093192
Signature: t1:5093192:1495161776:06e05544cff808b03546f753a553ecc66fafc79f
Summary:
When remotestore is a null store, uploading is a no-op and can be skipped.
This diff makes it so.
Test Plan: arc unit
Reviewers: #mercurial, davidsp
Reviewed By: davidsp
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5090686
Signature: t1:5090686:1495144964:956549307912f4a44201aa305959bf8de26ac028
This test had non-deterministic ordering, which caused it to fail sometimes.
Let's make the test check more explicitly, by checking that the bad commit hash
is not present.
Summary:
conduit HTTP POST request did not have correct content-type header set.
Now they do. Ideally we should use python-request package to do HTTP
but I do not know enough how to import these dependencies so I just went
with the minimal changes.
Test Plan:
ran hg log from dev environment and eventually got it all working.
{P57411895}
Reviewers: simpkins
Reviewed By: simpkins
Subscribers: net-systems-diffs@fb.com, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5080725
Tasks: 18294945
Signature: t1:5080725:1495052201:1bbd6edad4c1c4b3482a7479259460a815947630
Summary:
One of the heads may point to filtered commit and isbackedup command fails in
this case. Let's use repo.unfiltered()
Test Plan: arc unit
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D5001977
Signature: t1:5001977:1493907725:f6a138bd4e2cae48b64152fceaf73660ea91f43b
Summary: Infinitepush didn't clean up temp files during pulls.
Test Plan:
Choose any hg server that's not in vip (for example, hg014.lla2).
Run `hg pull ssh://hg014.lla2.facebook.com//data/scm/fbsource -r INFINITEPUSH_COMMIT_HASH`
Check that new /tmp/tmp* file has appeared.
Install fixed infinitepush extension on the server.
Run the same command again, make sure it works fine.
Note: no new unittests, because temp filenames are random and that's difficult to test.
Reviewers: #mercurial, dschatzberg, tja
Reviewed By: tja
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5077767
Tasks: 17599765
Signature: t1:5077767:1495025042:d209d1e5342442ac8b05a4c80748780bbf36846c
Summary:
`repair.strip` removes required revisions and their children unconditionally.
That means innocent children may be removed. This patch fixes it.
We may want to have an in-core function for this feature to not strip innocent
children.
Thanks Zibi Braniecki from Mozilla for reporting [1]!
[1]: https://bitbucket.org/facebook/hg-experimental/issues/6/hg-absorb-merges-diverged-commits
Test Plan: Added a test
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5073342
Signature: t1:5073342:1494972231:ca2a9881415119047771c8f35e059f4bfb1749e5
Summary:
remotefilelog needs to wait for changelog creation to get the commit hash so
all filelog appending operations are pending if linkrev is an integer.
That is better done in the `addrawrevision` layer since it will catch more code
paths. Besides, that means remotefilelog won't need to hash a same revision
twice.
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5061330
Signature: t1:5061330:1494871717:a7224ebdc0f221fbaabbd2a58de975caec0e4b05
Summary: This is required (but not complete) to allow LFS fast path.
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham
Differential Revision: https://phabricator.intern.facebook.com/D5061283
Signature: t1:5061283:1494959731:2d7ed7465c1724cd1f231a206f949da16f90649c
Summary: This makes the next patch cleaner to read.
Test Plan: arc unit
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5061271
Signature: t1:5061271:1494869273:967b647281847fa39e88558805bcf3b9a9e2b57b
Summary:
Core HG code uses revlog.nodemap to test node existence. We will hit some
code path about LFS in the future. So let's add a nodemap to remotefilelog.
Currently, the code path won't be hit. In the future, it should only be hit by
`repo._filecommit` when a `remotefilectx` is used (which is an LFS fast path).
That means, `nodemap` test won't connect remote server for missing nodes.
In the future, we could add some "hints" to get/getmeta API to let it not look
for the remote store.
Test Plan:
Real test will be added when we do can hit that code path. But the new code is
short and looks fine.
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: durham, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5061254
Signature: t1:5061254:1494869146:88a0e1d04d292e2a64f29fdf52660f48b906665c
Summary:
When a bundle is created, the bundle could be exchanged via copy and possibly
eventually reach to a publishing repo. If we don't upload LFS blobs, hg server
could have revisions that can never be checked out or verified.
So let's just assume bundles generated by `hg bundle` will be public, and
upload LFS blobs automatically, without depending on other code review tooling.
Note: there is a `preoutgoing` hook which will be triggered in this case,
however it's not useful since it does not have the `outgoing` information.
Test Plan: Will add a new test
Reviewers: davidsp, #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5050945
Signature: t1:5050945:1494958648:0c46ab1d85755838ba189bbb1e0673882922bb58
Summary:
`util.py` only contains 2 small things: `lfsvfs` and `sha256`. Both of them
only have one user (in terms of files). Therefore just move the code to
related files to make it simpler.
Test Plan: arc unit
Reviewers: davidsp, #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5067862
Signature: t1:5067862:1494955824:b99021f1bb44568dc0c738b0e472eb732f2bc91a
Summary:
The error was renamed to `InvalidPointer`. This diff updates error handling
to catch that instead.
Test Plan: arc unit
Reviewers: davidsp, #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5067496
Signature: t1:5067496:1494955510:5f0162a8cc6f1d0d83e3ab6319ec3202028684d8
Summary:
This simplifies code a bit, and could be useful for 3rd party code. We already
have `isbinary`, `islink`, `isexec`, `isabsent`. So another `is` method looks
fine.
Test Plan: Run existing tests
Reviewers: davidsp, #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5067369
Signature: t1:5067369:1494955466:ae17e310e743c704cc0d9bb73e0d2e82adcffff8
Summary:
The `filectx.cmp` fast path allows us to show "binary file changed" diff output
when two binaries have the same size but different content.
Test Plan: Added a test case
Reviewers: davidsp, #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5067266
Signature: t1:5067266:1494954783:a41e8213782afe24618ef2930c9576f21610fd3e
Summary:
When the server performs a repack, it would read all the data from all the
revlogs. This was very slow and expensive. Let's add a --incremental option that
makes it only read the revlog entries that have a linkrev newer than the latest
linkrev data that's already in a pack file.
Test Plan: Adds a test
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4997260
Signature: t1:4997260:1493904216:c4f5b6c9652bbf8f66c1bc2b2547c898324d22cd
Summary:
Calculating what blobs to upload should be fast since we use changelog `files`
information. Users will see `lfs: uploading ...` very soon. So let's downgrade
`lfs: computing set of blobs to upload` to a debug message to make `-v`
cleaner.
Test Plan: Updated existing test.
Reviewers: davidsp, #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5066600
Signature: t1:5066600:1494954115:c00925a1930f9b53e914078d32b6c4e4161099ee
Summary:
Downloading or uploading a single LFS object could take long. Instead of
showing a summary of what objects are downloaded or uploaded, print a message
per object if --verbose is provided.
Test Plan: Added a test
Reviewers: davidsp, #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5066327
Signature: t1:5066327:1494953894:3593436ff53eaff117a2932476dfc4b867fb9f17
Summary:
Previously, when an object is not found server-side, we error out during the
basic transfer API. This diffs move the check to the batch metadata API so the
check is preformed earlier.
That also means, when batch fetching many LFS objects, we could report multiple
objects being missing instead of just one.
Test Plan: Modified an existing test
Reviewers: davidsp, #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5065623
Signature: t1:5065623:1494952131:cb1df73e1cea21e07bfd0ec630bacec60e5b385c
Summary:
We are adding a simple transaction that works only with filelogs but
allows for concurrent access from multiple workers. This allows for a proper
rollback in case of a failure in a worker process, which previously would result
in bad data in the repositoriy.
Test Plan: rt test-p4* test-check*
Reviewers: #mercurial, durham, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5070340
Signature: t1:5070340:1494958313:b10b1eac5b42b36d1a587c4ae1c95fc2f8b5ad35
Summary:
The batch API allows server to return errors per object, handling them early
seems to be a good idea.
This is an attempt to distinguish "cannot upload - no permission" from
"no need to upload - already exist".
Unfortunately, `lfs-test-server` reference implementation does not provide
the error message so it's not reflected in tests.
Reviewers: #mercurial, davidsp, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5057816
Signature: t1:5057816:1494951879:07f2b67408e9d77255104b932baae2e50af9890b
Summary:
The LFS uploading or downloading process consists of 2 parts: a batch API to
get metadata about objects, followed by `len(objects)` requests.
This diff moves the first metadata API out from a giant `_batch` method to
make the code structure easier to maintain. Later diffs will further split
`_batch` method.
A side effect is `lfs: mapping blobs to #{action} URLs` message is removed.
More user-friendly message will be added back in a later patch.
Reviewers: #mercurial, davidsp, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5057541
Signature: t1:5057541:1494949989:d510c3d9369744e7ee776c0d0a26d263dfaf8352
Summary:
Displaying total bytes to upload is currently inaccurate - the server could
already have some objects so only a subset of selected objects will be
uploaded.
Besides, we pass `pointers` to related upload and download APIs so `total`
could be calculated from them.
This diff removes the inaccurate "need to upload" message and unnecessary
parameters. An accurate message will be added in a later patch.
Reviewers: #mercurial, davidsp, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5057470
Signature: t1:5057470:1494949554:07e637a8b58c894598b3f0c2dd53f80a5ade25a8
Summary:
Upstream shelve now writes simplekeyvaluefile for shelvedstate. This means
that tests which relied on ordered state files need to be adjusted.
Test Plan: - rt
Reviewers: durham, mitrandir, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D5069982
Summary: Added -U/--unified NUM to fbshow.py, and used that to set diff.unified=NUM.
Test Plan: hg show --help; hg show -U 1; hg show -U 2; hg show -U by induction.
Reviewers: mburman, simonfar, ikostia
Differential Revision: https://phabricator.intern.facebook.com/D5052156
Tasks: 10006036
Tags: python
Summary:
A try block is only effective for the HTTP error. Let's narrow it down to
just the HTTP request.
Test Plan: arc unit
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5057107
Signature: t1:5057107:1494634308:584877babc6a4705fa0740788dcf58affdf51bec
Summary:
There are `1+n` HTTP requests. `n` of them is handled. But the first one
about JSON metadata is not. Let's handle it.
Test Plan: arc unit
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5057053
Signature: t1:5057053:1494634184:fa0e562dc7879379106bf37915a2e5ea3cc01a4d
Summary:
The server or the HTTP library may provide more detailed error message.
Let's use them. `RequestFailedError` is also renamed to `LfsRemoteError` to
make it clear it's related to LFS and network.
Test Plan: arc unit
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5057014
Signature: t1:5057014:1494634052:00c1cb1b337a0e4cb92828cd49e95b7b0ff7e540
Summary:
It's caught, and re-raise with RequestFailedError. We can just raise
RequestFailedError directly.
Test Plan: arc unit
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5056886
Signature: t1:5056886:1494633307:3fdf2fab13e783224d343266a1278c41ed4394e8
Summary:
_batch is called by programmers. If action is not expected, it's a
ProgrammingError.
Test Plan: arc unit
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5056822
Signature: t1:5056822:1494634121:59737eb0d3f42d706ee8b0dc18ec7a4b49ada34f
Summary:
move p4fastimport under hgext3rd as that's where it belongs.
Also has the benefit that we package it up.
Test Plan: rt test-p4*
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5056664
Signature: t1:5056664:1494631758:4892f47922d8fbbd2c1f7793f3f29ee73fa8fa42
Summary:
Add a debugscanlfs command to p4fastimport. We want a similar command in
lfs but in this case we also want to do the reverse lookup of a
clientspec, etc.
Test Plan: $ unbuffer hg --config 'p4fastimport.useworker=False' debugscanlfs --debug --trace --client 'dsp-test' -A
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5055890
Signature: t1:5055890:1494629600:a20abf217abe51f26b179e0ebfdaf9230e3ad34d
Summary:
For long I have been wondering - we have `oid` already and an `oid` could
identify an object already, why do we need a `StoreID` which contains both
`oid` and `size`? It does not seem necessary?
I figured out the answer - the batch download or upload API *requires* size
information per object [1].
Since we will have LFS pointers during upload or download, we can just use
pointers. This diff removes `StoreID`, changes remote blob store to take
`pointers` directly, and local blob store to take `oid` directly.
As a side effect, `debuglfsupload` no longer takes `-o`, which seems fine
because `-r` should be more useful.
[1] https://github.com/git-lfs/git-lfs/blob/master/docs/api/batch.md
Test Plan: `arc unit`
Reviewers: davidsp, #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy
Differential Revision: https://phabricator.intern.facebook.com/D5043694
Signature: t1:5043694:1494496522:ea10f3dd12fc2ae3ab77b8f623eeead84dcc25fa
Summary:
Read the Git-LFS specification [1] and implement most checks to prevent
programming error and detect data corruption.
The new code should be stronger than what was before the refactoring, since the
old code only checks keys but not values.
[1]: https://github.com/git-lfs/git-lfs/blob/master/docs/spec.md
Test Plan: Added a new test
Reviewers: #mercurial, davidsp, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5046740
Signature: t1:5046740:1494543072:2f0dbfc07cd52a10572550a277d850bdf3f78d27
Summary:
Since we decided to only support GitHub's Git LFS specification, there is no
need to support multiple pointer types, so the code could be simplified.
The old code special case keys like `version`, `oid`, `hashalgo`, which makes
it longer than necessary. The new code is a rewrite treating everything as a
normal dict entry so the pointer class is much shorter: 76 -> 21 lines.
Data validation is temporary lost, which will be added back (and stronger) by
the next diff. It is separated to make review easier.
Test Plan: Run existing tests
Reviewers: #mercurial, davidsp, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy
Differential Revision: https://phabricator.intern.facebook.com/D5043547
Signature: t1:5043547:1494543031:ac1100939a10a79dfd749ab6ac9c3bb7fcd84dbf
Summary:
This diff adds a shared `p4setup.sh` that de-duplicates common logic among
tests. It also uses absolute path to make sure the extension being tested is
the version being developed.
The LFS test is also workarounded temporarily waiting for upstream change.
Test Plan: Run existing tests
Reviewers: #mercurial, davidsp
Reviewed By: davidsp
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5049279
Signature: t1:5049279:1494547832:28222fd2034115faca73860d6dd2f19206701aaa
Summary:
`rlog` is optional. This diff removes the following output from test output:
```
+ sh: rlog: command not found
```
Test Plan: Run existing test.
Reviewers: #mercurial, davidsp
Reviewed By: davidsp
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5047296
Signature: t1:5047296:1494535382:c57e554b57f28428a9d677d6e1089c81585e1bf2
Facebook probably doesn't want external users reaching out to them directly
for support, so this should be customizable to specify other possible courses
of action (especially in cases where githelp is further extended to document
git-related company-internal commands in other companies).
It looks like there is some flakeyness in racing pushes during this test. Seems
to be caused by hg serve loading the initial repo state while other processes
are writing new commits, which results in a missing master. Introducing a slight
sleep to the beginning of the transaction allows all the processes to load the
correct initial state and gets rid of the flakeyness.
Summary:
This hooks into the strip code to allow stripping revisions from treemanifests
as well. This will make it easier to clean up server repos when mistakes happen.
Test Plan: Adds a test
Reviewers: #mercurial
Differential Revision: https://phabricator.intern.facebook.com/D5031489
