Previously fastannotate was creating filectx's by doing commitctx[path]. This
invokes logic that resovles the filenode immediately, which require loading the
manifest for that commit. In a repo where manifests are downloaded lazily, this
can result in a lot of time spent downloading manifests.
Since commands like hg blame -u only need the filectx so they can resolve the
commitctx for commit information, let's just pass the commitctx straight to the
filectx. It can later derive the filenode if it needs to.
Differential Revision: https://phab.mercurial-scm.org/D1488
_fileid is only set in some cases. We should access the file node through
_filenode instead, which can compute the node from either the _fileid or the
_changeid. This will be useful in a future diff where we construct
remotefilelogctx with just a path and a commit hash, and not a file id.
This fixes the race condition in remotefilelog surfaced by
https://phab.mercurial-scm.org/D1458. The issue was that the remotefilelog
contentstore had 1 object for caching metadata of a file which could be
ovewriten by other threads, resulting in trying to
deserialize a textfile as lfs.
This adds per thread cachefor metadata
Test Plan:
on CentOS ran rt in fb-hgext and all were sucessful
on Windows ran 500 iterations of sparse --enable-profile / --disable-profile with 43k files
profile without hitting any issues. This was >30 hours of continuous excercise
for this code.
Differential Revision: https://phab.mercurial-scm.org/D1513
Previously, if a repo went from treeonly to not treeonly, the user had to run a
command to backfill the missing flat manifests. This patch makes it happen
automatically as part of hg pull.
Differential Revision: https://phab.mercurial-scm.org/D1485
Currently, normal LFS uploads are completely disabled during a p4fastimport,
and users must run an external SQLite-reading uploader app after the import.
Instead, allow normal LFS functionality to work, but users may override it to
do the SQLite thing.
Differential Revision: https://phab.mercurial-scm.org/D1307
p4fastimport has the ability to bypass the normal LFS upload mechanisms, and
that's what the original testcase actually tested.
We also want to ensure that normal LFS works with p4fastimport too, so we need
a testcase for that.
Differential Revision: https://phab.mercurial-scm.org/D1306
Summary:
Didn't work when --to wasn't specified as a 12 char
short hash, and since no errors were raised
due to the missing "raise" before error.abort
also effectively deleted changes in the working copy.
Now should work with anything repo[input] accepts
as a valid commit.
Also checks for predicatable error conditions before
the repo is modified to avoid changing working copy
on error.
Test Plan: TBD
Reviewers: #sourcecontrol
Subscribers: #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D6379590
Tasks: T22281996
The build currently fails on systems without Cython. Add a prebuilt cython
file for clindex to allow builds on those systems.
Differential Revision: https://phab.mercurial-scm.org/D1504
In `branchmap.updatecache`, there is an assertion:
assert partial.validfor(repo)
That will break if `partial` (branchcache) does not have correct tiprev or
tipnode.
The long term fix will be probably reviving D1450.
Summary:
D1451 changed _branchmapupdate() so that it no longer updates self.filteredhash
when changing self.tiprev. This causes branchcache.validfor() to report that
the cache is no longer valid, causing an assertion failure at the end of
updatecache().
Test Plan: Running hg commands no longer crash.
Reviewers: quark, durham, #fbhgext
Reviewed By: quark, #fbhgext
Differential Revision: https://phab.mercurial-scm.org/D1495
Summary: Fix lint.
Test Plan:
`test-check-code-hg.t` is now pasing
`test-check-config` is also passing
Reviewers: #fbhgext
Differential Revision: https://phab.mercurial-scm.org/D1491
Summary:
A better way to avoid deadlocks and not sacrifice performance on `_getfiles`
call.
Test Plan:
- build, pull and update on Windows
- build, pull and update on Linux
- do not observe it hanging
Reviewers: durham, #fbhgext
Differential Revision: https://phab.mercurial-scm.org/D1467
Summary: We're migrating towards the new tech.
Test Plan: tried deleting 'cert' from my .arcrc, still works
Reviewers: medson, #mercurial
Reviewed By: medson
Subscribers: medson, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D6376729
Signature: 6376729:1511302323:24066afd17b14c13100b70df9c0fca9220e71799
Summary:
The tests `test-check-pyflakes-hg.t` and `test-check-code-hg.t` are
currently failing because of this.
Test Plan: Ran all the tests.
Reviewers: #fbhgext, quark
Reviewed By: #fbhgext, quark
Differential Revision: https://phab.mercurial-scm.org/D1489
This patch did two micro optimizations:
- Avoid sorting `headrevs` since it's already sorted.
- Inline `cl.node` so there is no `node` hash table lookups inside the loop.
These are good practices. Although practically we don't have that many
headrevs to notice a difference.
Differential Revision: https://phab.mercurial-scm.org/D1452
filteredhash, tiprev, tipnode are only used for detecting whether the
on-disk cache is up-to-date or not. Since we don't have on-disk cache, it's
unnecessary to calculate them.
Differential Revision: https://phab.mercurial-scm.org/D1451
The statefile-not-set case was not tested. Previously the code will make
`ui._progbar` return None, which means no progress bar.
Also move the docstring to the header so it shows up in `hg help -e`, and
use modern config registrar to avoid devel warnings.
Test Plan:
Added a test
Differential Revision: https://phab.mercurial-scm.org/D1486
Previously, once you transitioned to treeonly mode you had no way of switching
back to hybrid mode. This patch adds a hg backfillmanifestrevlog command to
refill a manifest revlog with any bits that are missing.
Differential Revision: https://phab.mercurial-scm.org/D1456
The manifestctx constructor changed at some point in the past to take a
manifestlog and a node instead of a repo and a node. The pushrebase cache code
wasn't adjusted for this. Luckily the manifestctx object only ever uses the
manifestlog to look at the revlog, and the revlog is only ever used to look at
the delta as a fastpath, so most code paths weren't affected.
We encountered this issue on our server, despite it being in there for several
months. Unfortunately I wasn't able to repro it in a test, but I did insert
manual manifestctx._revlog() lines after the construction to ensure that the
revlog can now be created, versus crashing before.
Differential Revision: https://phab.mercurial-scm.org/D1439
Previously hg pull created it's own ssh connection, then the tree prefetch
created one as well. Let's change hg pull to use a connection from the pool when
possible, so it can be reused by treemanifest later.
Differential Revision: https://phab.mercurial-scm.org/D1455
remotefilelog and fastannotate already use a connection pool to share and reuse
connections. Treemanifest often does ondemand downloading of trees, such as
during hg log -p, and would greatly benefit from reusing connections as well.
This patch makes the connectionpool and attribute of the repo object, instead of
the fileserverclient object, which allows treemanifest to make use of it easily.
Differential Revision: https://phab.mercurial-scm.org/D1454
Summary:
This allows us to worry less about deadlocks and be more efficient in our
piped communications (like in fileserverclient.py, for example).
We can also make sure that deadlocks just plainly can't
happen by only writing a known amount of bytes to the pipe.
Test Plan:
- does not seem to break any additional tests on Linux
Reviewers: #fbhgext
Differential Revision: https://phab.mercurial-scm.org/D1436
Previously sparse will resolve working file context via its parent
unconditionally if the file is outside sparsematch (introduced by D788).
That could be problematic if the file only exists in working copy.
This patch changes it to always try working copy first (fast), then fallback
to parent commit if the file is outside sparse.
Differential Revision: https://phab.mercurial-scm.org/D1464
Summary:
This is a follow up to D1446. It extends the prefetch logic to not
download trees even for commits with `secret` phase. Also, it uses a better way
of dealing with revsets. Thanks @quark for this!
Test Plan: Ran all the tests.
Reviewers: #fbhgext, quark
Reviewed By: #fbhgext, quark
Subscribers: quark
Differential Revision: https://phab.mercurial-scm.org/D1461
osutil.listdir yields (filename, type, stat), not (filename, size, stat), so we
need to look at the stat.st_size value to get the size. (Previously, we were
summing the file type codes :/)
This should fix packfile metrics.
Differential Revision: https://phab.mercurial-scm.org/D1447
Computing the hidden commits was showing up as a hot spot in hg status because
hg sparse was accessing the changelog through a filtered repo. This probably
affects many other commands since sparse touches so many places. Since it has no
need to use a filtered repo, let's just have it use unfiltered.
This shaved off 25-30% of the hg status time in some circumstances.
Differential Revision: https://phab.mercurial-scm.org/D1437
Summary:
After D1417, `hg prefetch` takes care of downloading both the files
and trees during the prefetch. However, when the command is run without any
options, it attempts to prefetch the trees for the draft commits which results
in an error. We should not even attempt to prefetch trees for the draft
commits.
Test Plan: Added a test to detect this case and ran all the tests.
Reviewers: #fbhgext, durham
Reviewed By: #fbhgext, durham
Subscribers: durham
Differential Revision: https://phab.mercurial-scm.org/D1446
This extension is a thin wrapper around the native `changelog.index` object
that allows us to incrementally replace its methods. Since some index
methods (ex. `nodemap.__getitem__`) are called very frequently, Cython
features are used intentionally to avoid overhead. This also makes it easier
to integrate logic with C interface in the future.
As a side effect, this patch enforce `index` to be conceptually separate
from `nodemap`. So `changelog.index[node]` could be made illegal in the
future, which seems to be a good thing.
Test Plan:
Run `hg sl` with and without the extension in a large repo. Check traceprof
outputs. Notice the performance difference around index methods are roughly
10%, which seems acceptable:
Without the extension:
25 \ node (4823 times) changelog.py:361
18 | node (4931 times) revlog.py:631
With the extension:
27 \ node (4823 times) changelog.py:361
19 | node (4931 times) revlog.py:631
Also run `rt --extra-config-opt=extensions.clindex=` from core hg and make
sure changes are all caused by having an extra extension enabled.
Differential Revision: https://phab.mercurial-scm.org/D1353
The number 200 was used before D1435. It caused trouble on systems with low
`ulimit -n` and with the Python datapack code path because Python's mmap
implementation keeps an internal fd for every mmap object and there is no
way to close those fds via pure Python API. But there is no such limit for
cdatapack after D1185. So let's change cdatapack test to use 200 packs.
Test Plan:
`ulimit -n 50` and `./scripts/unit.py`
Differential Revision: https://phab.mercurial-scm.org/D1442
Summary:
Currently,
- `hg prefetch` prefetches files.
- `hg prefetchtrees` prefetches trees.
This commit removes `prefetchtrees` and makes `prefetch` responsible for
everything i.e. `prefetch` will prefetch whatever it can prefetch be it files,
trees, or both.
Test Plan: Ran all the tests.
Reviewers: #fbhgext, durham
Reviewed By: #fbhgext, durham
Subscribers: quark, durham
Differential Revision: https://phab.mercurial-scm.org/D1417
Summary:
The `prefetch` command has an option to repack the prefetched files.
Eventually, we plan to merge `prefetch` and `prefetchtrees` into a single
command and therefore, this commit takes a step towards making the interface to
these commands exactly the same.
Test Plan: Ran all the tests.
Reviewers: #fbhgext, durham
Reviewed By: #fbhgext, durham
Subscribers: durham
Differential Revision: https://phab.mercurial-scm.org/D1416
Summary:
The `prefetch` command only allows for one revision to be specified as
the base revision. Eventually, we plan to merge `prefetch` and `prefetchtrees`
into a single command and therefore, this commit takes a step towards making
the interface to these commands exactly the same.
Test Plan: Ran all the tests.
Reviewers: #fbhgext, durham
Reviewed By: #fbhgext, durham
Differential Revision: https://phab.mercurial-scm.org/D1415
Summary:
The `prefetch` command performs a preprocessing of the options before
doing the actual work. This commit just separates out that logic.
Test Plan: Ran all the tests.
Reviewers: #fbhgext, durham
Reviewed By: #fbhgext, durham
Subscribers: durham
Differential Revision: https://phab.mercurial-scm.org/D1414
Summary:
Adding the option to specify the base revision in the `prefetch`
command. This can useful to limit the prefetch data and also makes the
interface of `prefetch` consistent with `prefetchtrees`. Soon, we will merge
`prefetch` and `prefetchtrees` into a single command and both commands having a
similar interface is useful for the merging.
Test Plan: Ran all the tests.
Reviewers: #fbhgext, durham
Reviewed By: #fbhgext, durham
Subscribers: durham
Differential Revision: https://phab.mercurial-scm.org/D1368
Summary: Let the `prefetch` command be responsible for the repacking.
Test Plan: Ran all the tests.
Reviewers: #fbhgext, durham
Reviewed By: #fbhgext, durham
Differential Revision: https://phab.mercurial-scm.org/D1367
The exec bit got lost after rebase.
The rebase bug was filed as https://bz.mercurial-scm.org/5743.
The file content change is to workaround a potential pushrebase bug that
does not allow mode-only change.
Previously the test sets up `LD_LIBRARY_PATH` and `PYTHONPATH`, then runs
Python tests.
Within Python code, setting `sys.path` would achieve the same effect of
setting `PYTHONPATH`. For `LD_LIBRARY_PATH`, it's necessary for C libraries.
But the only C library that cstore depends on is `lz4`, which is supposed to
use the system version. There is no C library provided by this repo -
features like sha1 are compiled in `cstore.so`.
Therefore it's unnecessary to have a separate `.t` file wrapping `.py`
tests. Let's just use `.py` tests directly.
Test Plan:
`./script/unit.py`
Make a temporary change to `cdatapack.c` so it fails unconditionally in
open_datapack. Build the repo in different ways: `make local` and
`python2 setup.py build_clib build_ext`. Then run the test by using
`$HG_CREW/tests/run-tests.py -l test-remotefilelog-datapack.py` without the
`hg-dev` environment and make sure it fails with the expected exception.
Differential Revision: https://phab.mercurial-scm.org/D1429
If the os limited a given process to a <200 files open, this test would fail.
Let's change the cache size to be smaller to avoid this.
Also, it turns out the cache size and number of packs created doesn't actually
seem to affect this test. I changed the numbers in a few ways and the test never
failed.
Differential Revision: https://phab.mercurial-scm.org/D1435
Added workers in lfs.
I had to remove the fine progress tracking because between processes in *nix and threads in windows (diffs will appear soon) the tracking of 1MB progress is quite tricky.
With our network tracking progress per file is way enough to see things moving.
This change gives close to 50% speedup on hg sparse --enable-profile when prefetch is run. My current understanding is that prefetch is ran when profile is enabled for the first time.
Test Plan:
Enable profile:
time hg sprase --enable-profile SparseProfiles/TestProfile.sparse
The profile contains 42k files including 9GB of lfs files
On my machine the time improves by 47% while still being dominated by lfs
download time
# Tip: In Git and Mercurial, use a branch like "T123" to automatically associate
# changes with the corresponding task.
Differential Revision: https://phab.mercurial-scm.org/D1424
