Treedirstate maps won't have an inner _map, so will never be logged by
perftweaks. The maps have an efficient size lookup, so it's OK to always
obtain their size.
Differential Revision: https://phab.mercurial-scm.org/D1410
The dirfoldmap and filefoldmap on the dirstatemap object map from normalized
filename to the form it appears in the dirstate (denormalized form).
The dicts these functions return are also modified by the dirstate with
filenames discovered by examining the disk, so use an overlay dictionary backed
by the real dirstate to allow this to happen.
Differential Revision: https://phab.mercurial-scm.org/D1409
Change to use VLQ-encoded numbers for everything in the tree file. Block sizes
remain as u32s so that they can be read by the store in a single read
operation, but everything else is a VLQ as it is generally smaller and more
futureproof.
Differential Revision: https://phab.mercurial-scm.org/D1408
Add a mechanism to the dirstate trees to allow lookups based on filtered views
of the keys. For a given filtering function, this returns one (if any) of the
keys for which filter(key) matches the input. The filtered values in each
directory node are cached to improve subsequent lookups.
Differential Revision: https://phab.mercurial-scm.org/D1406
If the non-normal sets get particularly large, the dirstate root file can get
large again. Avoid this happening by not storing large sets, and instead
recalculating them on-demand as needed. Large sets of non-normal files should
be rare, so this shouldn't be a performance hit in the normal case.
Differential Revision: https://phab.mercurial-scm.org/D1405
When writing out changes to the dirstate, mark any files that have the same
mtime as the current time as requiring lookup. This matches the behaviour
of the C-based pack_dirstate.
Differential Revision: https://phab.mercurial-scm.org/D1404
In order to allow the python interface code to perform actions on each node in
the tree without creating python-specific interfaces in the generic Rust code,
add a method of iterating over the tree, executing a closure at each file.
Use this to implement the methods that give iterators over the filenames in the
tree. This performs better than the get_first/get_next-style iterators.
Differential Revision: https://phab.mercurial-scm.org/D1403
Iterations over all files will cause most of the file to be read in a piecemeal
fashion. This will be inefficient on disks with slow seek times. Instead,
read the whole file into memory before iterating.
Differential Revision: https://phab.mercurial-scm.org/D1402
Adds FileStore, an implementation of the Store and StoreView traits that uses a
file on disk to store the data, and reads and writes blocks using file I/O.
Differential Revision: https://phab.mercurial-scm.org/D1398
These traits represent abstract store objects than can store arbitrary data
blocks with store-generated indexes.
A NullStore implementation is provided which acts an always-empty StoreView.
Differential Revision: https://phab.mercurial-scm.org/D1397
This adds an implementation of an ordered map that uses a vector pairs, sorted
by the key.
This is largely compatibly with std::collections::BTreeMap, but has performance
characteristics more suited for use in treedirstate.
Differential Revision: https://phab.mercurial-scm.org/D1396
Create an empty Rust project for treedirstate. This will be a
re-implementation of the dirstate map using a tree structure, where nodes in
the tree are directories, and leaves are files.
Differential Revision: https://phab.mercurial-scm.org/D1395
Previously fastannotate was creating filectx's by doing commitctx[path]. This
invokes logic that resovles the filenode immediately, which require loading the
manifest for that commit. In a repo where manifests are downloaded lazily, this
can result in a lot of time spent downloading manifests.
Since commands like hg blame -u only need the filectx so they can resolve the
commitctx for commit information, let's just pass the commitctx straight to the
filectx. It can later derive the filenode if it needs to.
Differential Revision: https://phab.mercurial-scm.org/D1488
_fileid is only set in some cases. We should access the file node through
_filenode instead, which can compute the node from either the _fileid or the
_changeid. This will be useful in a future diff where we construct
remotefilelogctx with just a path and a commit hash, and not a file id.
This fixes the race condition in remotefilelog surfaced by
https://phab.mercurial-scm.org/D1458. The issue was that the remotefilelog
contentstore had 1 object for caching metadata of a file which could be
ovewriten by other threads, resulting in trying to
deserialize a textfile as lfs.
This adds per thread cachefor metadata
Test Plan:
on CentOS ran rt in fb-hgext and all were sucessful
on Windows ran 500 iterations of sparse --enable-profile / --disable-profile with 43k files
profile without hitting any issues. This was >30 hours of continuous excercise
for this code.
Differential Revision: https://phab.mercurial-scm.org/D1513
Previously, if a repo went from treeonly to not treeonly, the user had to run a
command to backfill the missing flat manifests. This patch makes it happen
automatically as part of hg pull.
Differential Revision: https://phab.mercurial-scm.org/D1485
Currently, normal LFS uploads are completely disabled during a p4fastimport,
and users must run an external SQLite-reading uploader app after the import.
Instead, allow normal LFS functionality to work, but users may override it to
do the SQLite thing.
Differential Revision: https://phab.mercurial-scm.org/D1307
p4fastimport has the ability to bypass the normal LFS upload mechanisms, and
that's what the original testcase actually tested.
We also want to ensure that normal LFS works with p4fastimport too, so we need
a testcase for that.
Differential Revision: https://phab.mercurial-scm.org/D1306
Summary:
Didn't work when --to wasn't specified as a 12 char
short hash, and since no errors were raised
due to the missing "raise" before error.abort
also effectively deleted changes in the working copy.
Now should work with anything repo[input] accepts
as a valid commit.
Also checks for predicatable error conditions before
the repo is modified to avoid changing working copy
on error.
Test Plan: TBD
Reviewers: #sourcecontrol
Subscribers: #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D6379590
Tasks: T22281996
The build currently fails on systems without Cython. Add a prebuilt cython
file for clindex to allow builds on those systems.
Differential Revision: https://phab.mercurial-scm.org/D1504
In `branchmap.updatecache`, there is an assertion:
assert partial.validfor(repo)
That will break if `partial` (branchcache) does not have correct tiprev or
tipnode.
The long term fix will be probably reviving D1450.
Summary:
D1451 changed _branchmapupdate() so that it no longer updates self.filteredhash
when changing self.tiprev. This causes branchcache.validfor() to report that
the cache is no longer valid, causing an assertion failure at the end of
updatecache().
Test Plan: Running hg commands no longer crash.
Reviewers: quark, durham, #fbhgext
Reviewed By: quark, #fbhgext
Differential Revision: https://phab.mercurial-scm.org/D1495
Summary: Fix lint.
Test Plan:
`test-check-code-hg.t` is now pasing
`test-check-config` is also passing
Reviewers: #fbhgext
Differential Revision: https://phab.mercurial-scm.org/D1491
Summary:
A better way to avoid deadlocks and not sacrifice performance on `_getfiles`
call.
Test Plan:
- build, pull and update on Windows
- build, pull and update on Linux
- do not observe it hanging
Reviewers: durham, #fbhgext
Differential Revision: https://phab.mercurial-scm.org/D1467
Summary: We're migrating towards the new tech.
Test Plan: tried deleting 'cert' from my .arcrc, still works
Reviewers: medson, #mercurial
Reviewed By: medson
Subscribers: medson, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D6376729
Signature: 6376729:1511302323:24066afd17b14c13100b70df9c0fca9220e71799
Summary:
The tests `test-check-pyflakes-hg.t` and `test-check-code-hg.t` are
currently failing because of this.
Test Plan: Ran all the tests.
Reviewers: #fbhgext, quark
Reviewed By: #fbhgext, quark
Differential Revision: https://phab.mercurial-scm.org/D1489
This patch did two micro optimizations:
- Avoid sorting `headrevs` since it's already sorted.
- Inline `cl.node` so there is no `node` hash table lookups inside the loop.
These are good practices. Although practically we don't have that many
headrevs to notice a difference.
Differential Revision: https://phab.mercurial-scm.org/D1452
filteredhash, tiprev, tipnode are only used for detecting whether the
on-disk cache is up-to-date or not. Since we don't have on-disk cache, it's
unnecessary to calculate them.
Differential Revision: https://phab.mercurial-scm.org/D1451
The statefile-not-set case was not tested. Previously the code will make
`ui._progbar` return None, which means no progress bar.
Also move the docstring to the header so it shows up in `hg help -e`, and
use modern config registrar to avoid devel warnings.
Test Plan:
Added a test
Differential Revision: https://phab.mercurial-scm.org/D1486
Previously, once you transitioned to treeonly mode you had no way of switching
back to hybrid mode. This patch adds a hg backfillmanifestrevlog command to
refill a manifest revlog with any bits that are missing.
Differential Revision: https://phab.mercurial-scm.org/D1456
The manifestctx constructor changed at some point in the past to take a
manifestlog and a node instead of a repo and a node. The pushrebase cache code
wasn't adjusted for this. Luckily the manifestctx object only ever uses the
manifestlog to look at the revlog, and the revlog is only ever used to look at
the delta as a fastpath, so most code paths weren't affected.
We encountered this issue on our server, despite it being in there for several
months. Unfortunately I wasn't able to repro it in a test, but I did insert
manual manifestctx._revlog() lines after the construction to ensure that the
revlog can now be created, versus crashing before.
Differential Revision: https://phab.mercurial-scm.org/D1439
Previously hg pull created it's own ssh connection, then the tree prefetch
created one as well. Let's change hg pull to use a connection from the pool when
possible, so it can be reused by treemanifest later.
Differential Revision: https://phab.mercurial-scm.org/D1455
remotefilelog and fastannotate already use a connection pool to share and reuse
connections. Treemanifest often does ondemand downloading of trees, such as
during hg log -p, and would greatly benefit from reusing connections as well.
This patch makes the connectionpool and attribute of the repo object, instead of
the fileserverclient object, which allows treemanifest to make use of it easily.
Differential Revision: https://phab.mercurial-scm.org/D1454
Summary:
This allows us to worry less about deadlocks and be more efficient in our
piped communications (like in fileserverclient.py, for example).
We can also make sure that deadlocks just plainly can't
happen by only writing a known amount of bytes to the pipe.
Test Plan:
- does not seem to break any additional tests on Linux
Reviewers: #fbhgext
Differential Revision: https://phab.mercurial-scm.org/D1436
Previously sparse will resolve working file context via its parent
unconditionally if the file is outside sparsematch (introduced by D788).
That could be problematic if the file only exists in working copy.
This patch changes it to always try working copy first (fast), then fallback
to parent commit if the file is outside sparse.
Differential Revision: https://phab.mercurial-scm.org/D1464
Summary:
This is a follow up to D1446. It extends the prefetch logic to not
download trees even for commits with `secret` phase. Also, it uses a better way
of dealing with revsets. Thanks @quark for this!
Test Plan: Ran all the tests.
Reviewers: #fbhgext, quark
Reviewed By: #fbhgext, quark
Subscribers: quark
Differential Revision: https://phab.mercurial-scm.org/D1461
osutil.listdir yields (filename, type, stat), not (filename, size, stat), so we
need to look at the stat.st_size value to get the size. (Previously, we were
summing the file type codes :/)
This should fix packfile metrics.
Differential Revision: https://phab.mercurial-scm.org/D1447
Computing the hidden commits was showing up as a hot spot in hg status because
hg sparse was accessing the changelog through a filtered repo. This probably
affects many other commands since sparse touches so many places. Since it has no
need to use a filtered repo, let's just have it use unfiltered.
This shaved off 25-30% of the hg status time in some circumstances.
Differential Revision: https://phab.mercurial-scm.org/D1437
