Moving the lock into the shared cache will allow multiple copies of the same
repository to avoid starting repacks at the same time.
Differential Revision: https://phab.mercurial-scm.org/D1544
Previously repack use the standard Mercurial symlink based locking mechanism.
This caused problems on our laptop users because the symlink locking relies on
the host name and sometimes their hostname changes due to weird IT issues, which
resulted in locks existing forever and repack never running. The symlink based
locking scheme was also a problem in chroots, where two processes in different
chroots may attempt to repack the same shared cache at the same time.
Switching to a fcntllock based scheme will solve these issues.
Differential Revision: https://phab.mercurial-scm.org/D1543
Summary:
If no remote path is configured raise an Abort exception, rather than returning
the path as None and crashing later on with an unfriendly exception due to
trying to use None as a string.
Test Plan: Updated test-treemanifest-prefetch.t to exercise this code.
Reviewers: singhsrb, durham, #fbhgext
Reviewed By: durham, #fbhgext
Differential Revision: https://phab.mercurial-scm.org/D1541
Summary:
Rename `_prefetchtrees()` to `prefetchtrees()`. Also change it to throw a
new `MissingNodesError()` exception type when it fails to download some nodes.
This makes it possible for callers to easily distinguish missing nodes from
other unexpected errors.
Test Plan:
Confirmed the existing unit tests pass. test-treemanifest-infinitepush.t
exercises the MissingNodesError code path.
Reviewers: #fbhgext, durham
Reviewed By: #fbhgext, durham
Subscribers: singhsrb
Differential Revision: https://phab.mercurial-scm.org/D1421
Summary:
When failing due to attempting to rebase already-public commits, include the
public commit node IDs in the error message.
Test Plan: Included new tests.
Reviewers: durham, quark, #fbhgext
Reviewed By: durham, quark, #fbhgext
Differential Revision: https://phab.mercurial-scm.org/D1496
This adds initialization of metacache in remaining place where it can be uninitalized when using threads
The previous state did fail on some hg update operations on Windows. It didn't
affect posix as that doesn't actually use threads so we always have metacache
initalized from the constructor
Test Plan:
Run hg update that failed wihtout this change on windows and see it
suceed.
run tests on CentOS
Differential Revision: https://phab.mercurial-scm.org/D1539
D1410 added support for treedirstate when collecting dirstate_size for
sampling. However, it also changed from querying the inner dirstate map (which
is not available with treedirstate) to the outer map. This breaks Eden, which
prevents calls to `__len__` on the outer map.
Revert the change for non-treedirstate dirstates, and handle treedirstate
explicitly.
Differential Revision: https://phab.mercurial-scm.org/D1530
To avoid problems when other dirstate implementations are in use (sqldirstate
or eden), prevent upgrade of repos with those implementations active to
treedirstate.
Automatic upgrades are silently prevented. Attempts to manually upgrade fail
with an explanatory message.
Differential Revision: https://phab.mercurial-scm.org/D1528
Remove the type parameter from Dirstate. It's not necessary, and complicates
the implementation.
Differential Revision: https://phab.mercurial-scm.org/D1512
The serialization and deserialization of the treedirstate tree file is
scattered across various functions in several files. Gather these together in
a single module so that the file format is obvious.
Differential Revision: https://phab.mercurial-scm.org/D1510
Copy the dirstate tests from core Mercurial, but run them with treedirstate
instead of the default dirstate map.
Add an extra test that covers scenarios specific to treedirstate.
Differential Revision: https://phab.mercurial-scm.org/D1490
The treedirstate and rusttreedirstate packages are only built if a suitable
version of Cargo is available.
Differential Revision: https://phab.mercurial-scm.org/D1412
Treedirstate maps won't have an inner _map, so will never be logged by
perftweaks. The maps have an efficient size lookup, so it's OK to always
obtain their size.
Differential Revision: https://phab.mercurial-scm.org/D1410
The dirfoldmap and filefoldmap on the dirstatemap object map from normalized
filename to the form it appears in the dirstate (denormalized form).
The dicts these functions return are also modified by the dirstate with
filenames discovered by examining the disk, so use an overlay dictionary backed
by the real dirstate to allow this to happen.
Differential Revision: https://phab.mercurial-scm.org/D1409
Change to use VLQ-encoded numbers for everything in the tree file. Block sizes
remain as u32s so that they can be read by the store in a single read
operation, but everything else is a VLQ as it is generally smaller and more
futureproof.
Differential Revision: https://phab.mercurial-scm.org/D1408
Add a mechanism to the dirstate trees to allow lookups based on filtered views
of the keys. For a given filtering function, this returns one (if any) of the
keys for which filter(key) matches the input. The filtered values in each
directory node are cached to improve subsequent lookups.
Differential Revision: https://phab.mercurial-scm.org/D1406
If the non-normal sets get particularly large, the dirstate root file can get
large again. Avoid this happening by not storing large sets, and instead
recalculating them on-demand as needed. Large sets of non-normal files should
be rare, so this shouldn't be a performance hit in the normal case.
Differential Revision: https://phab.mercurial-scm.org/D1405
When writing out changes to the dirstate, mark any files that have the same
mtime as the current time as requiring lookup. This matches the behaviour
of the C-based pack_dirstate.
Differential Revision: https://phab.mercurial-scm.org/D1404
In order to allow the python interface code to perform actions on each node in
the tree without creating python-specific interfaces in the generic Rust code,
add a method of iterating over the tree, executing a closure at each file.
Use this to implement the methods that give iterators over the filenames in the
tree. This performs better than the get_first/get_next-style iterators.
Differential Revision: https://phab.mercurial-scm.org/D1403
Iterations over all files will cause most of the file to be read in a piecemeal
fashion. This will be inefficient on disks with slow seek times. Instead,
read the whole file into memory before iterating.
Differential Revision: https://phab.mercurial-scm.org/D1402
Adds FileStore, an implementation of the Store and StoreView traits that uses a
file on disk to store the data, and reads and writes blocks using file I/O.
Differential Revision: https://phab.mercurial-scm.org/D1398
These traits represent abstract store objects than can store arbitrary data
blocks with store-generated indexes.
A NullStore implementation is provided which acts an always-empty StoreView.
Differential Revision: https://phab.mercurial-scm.org/D1397
This adds an implementation of an ordered map that uses a vector pairs, sorted
by the key.
This is largely compatibly with std::collections::BTreeMap, but has performance
characteristics more suited for use in treedirstate.
Differential Revision: https://phab.mercurial-scm.org/D1396
Create an empty Rust project for treedirstate. This will be a
re-implementation of the dirstate map using a tree structure, where nodes in
the tree are directories, and leaves are files.
Differential Revision: https://phab.mercurial-scm.org/D1395
Previously fastannotate was creating filectx's by doing commitctx[path]. This
invokes logic that resovles the filenode immediately, which require loading the
manifest for that commit. In a repo where manifests are downloaded lazily, this
can result in a lot of time spent downloading manifests.
Since commands like hg blame -u only need the filectx so they can resolve the
commitctx for commit information, let's just pass the commitctx straight to the
filectx. It can later derive the filenode if it needs to.
Differential Revision: https://phab.mercurial-scm.org/D1488
_fileid is only set in some cases. We should access the file node through
_filenode instead, which can compute the node from either the _fileid or the
_changeid. This will be useful in a future diff where we construct
remotefilelogctx with just a path and a commit hash, and not a file id.
The rule only triggered on non Windows platforms, even though Windows also
required an adjustment. Automatic seems better.
The aggressive globbing in test-subrepo-svn.t was found and rewritten by the
substitution.
Summary:
This is way more common than we thought, and these files aren't actually useful
at the moment. Let's drop them so they don't pollute the filesystem.
Test Plan: Ran the tests
Reviewers: quark, #mercurial
Reviewed By: quark
Differential Revision: https://phabricator.intern.facebook.com/D6420066
Signature: 6420066:1511826415:386036ddbded31fed7892cdce48b9b38f535e915
This fixes the race condition in remotefilelog surfaced by
https://phab.mercurial-scm.org/D1458. The issue was that the remotefilelog
contentstore had 1 object for caching metadata of a file which could be
ovewriten by other threads, resulting in trying to
deserialize a textfile as lfs.
This adds per thread cachefor metadata
Test Plan:
on CentOS ran rt in fb-hgext and all were sucessful
on Windows ran 500 iterations of sparse --enable-profile / --disable-profile with 43k files
profile without hitting any issues. This was >30 hours of continuous excercise
for this code.
Differential Revision: https://phab.mercurial-scm.org/D1513
Previously, if a repo went from treeonly to not treeonly, the user had to run a
command to backfill the missing flat manifests. This patch makes it happen
automatically as part of hg pull.
Differential Revision: https://phab.mercurial-scm.org/D1485
Currently, normal LFS uploads are completely disabled during a p4fastimport,
and users must run an external SQLite-reading uploader app after the import.
Instead, allow normal LFS functionality to work, but users may override it to
do the SQLite thing.
Differential Revision: https://phab.mercurial-scm.org/D1307
p4fastimport has the ability to bypass the normal LFS upload mechanisms, and
that's what the original testcase actually tested.
We also want to ensure that normal LFS works with p4fastimport too, so we need
a testcase for that.
Differential Revision: https://phab.mercurial-scm.org/D1306
Summary:
Didn't work when --to wasn't specified as a 12 char
short hash, and since no errors were raised
due to the missing "raise" before error.abort
also effectively deleted changes in the working copy.
Now should work with anything repo[input] accepts
as a valid commit.
Also checks for predicatable error conditions before
the repo is modified to avoid changing working copy
on error.
Test Plan: TBD
Reviewers: #sourcecontrol
Subscribers: #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D6379590
Tasks: T22281996
In large repositories, updates involving the creation of many files check the
same directories repeatedly in the wctx manifest. Move these checks out to a
separate loop to avoid repeated checks hitting the manifest.
Differential Revision: https://phab.mercurial-scm.org/D1226
As mentioned in D1222, the recent pathconflicts change regresses update
performance in large repositories when many files are being updated.
To mitigate this, we introduce two caches of directories that have
already found to be either:
- unknown directories, but which are not aliased by files and
so don't need to be checked if they are files again; and
- missing directores, which cannot cause path conflicts, and
cannot contain a file that causes a path conflict.
When checking the paths of a file, testing against this caches means we can
skip tests that involve touching the filesystem.
Differential Revision: https://phab.mercurial-scm.org/D1224
The build currently fails on systems without Cython. Add a prebuilt cython
file for clindex to allow builds on those systems.
Differential Revision: https://phab.mercurial-scm.org/D1504
In `branchmap.updatecache`, there is an assertion:
assert partial.validfor(repo)
That will break if `partial` (branchcache) does not have correct tiprev or
tipnode.
The long term fix will be probably reviving D1450.
