Previously sparse will resolve working file context via its parent
unconditionally if the file is outside sparsematch (introduced by D788).
That could be problematic if the file only exists in working copy.
This patch changes it to always try working copy first (fast), then fallback
to parent commit if the file is outside sparse.
Differential Revision: https://phab.mercurial-scm.org/D1464
Summary:
After D1417, `hg prefetch` takes care of downloading both the files
and trees during the prefetch. However, when the command is run without any
options, it attempts to prefetch the trees for the draft commits which results
in an error. We should not even attempt to prefetch trees for the draft
commits.
Test Plan: Added a test to detect this case and ran all the tests.
Reviewers: #fbhgext, durham
Reviewed By: #fbhgext, durham
Subscribers: durham
Differential Revision: https://phab.mercurial-scm.org/D1446
The number 200 was used before D1435. It caused trouble on systems with low
`ulimit -n` and with the Python datapack code path because Python's mmap
implementation keeps an internal fd for every mmap object and there is no
way to close those fds via pure Python API. But there is no such limit for
cdatapack after D1185. So let's change cdatapack test to use 200 packs.
Test Plan:
`ulimit -n 50` and `./scripts/unit.py`
Differential Revision: https://phab.mercurial-scm.org/D1442
Summary:
Currently,
- `hg prefetch` prefetches files.
- `hg prefetchtrees` prefetches trees.
This commit removes `prefetchtrees` and makes `prefetch` responsible for
everything i.e. `prefetch` will prefetch whatever it can prefetch be it files,
trees, or both.
Test Plan: Ran all the tests.
Reviewers: #fbhgext, durham
Reviewed By: #fbhgext, durham
Subscribers: quark, durham
Differential Revision: https://phab.mercurial-scm.org/D1417
Summary:
The `prefetch` command has an option to repack the prefetched files.
Eventually, we plan to merge `prefetch` and `prefetchtrees` into a single
command and therefore, this commit takes a step towards making the interface to
these commands exactly the same.
Test Plan: Ran all the tests.
Reviewers: #fbhgext, durham
Reviewed By: #fbhgext, durham
Subscribers: durham
Differential Revision: https://phab.mercurial-scm.org/D1416
Summary:
Adding the option to specify the base revision in the `prefetch`
command. This can useful to limit the prefetch data and also makes the
interface of `prefetch` consistent with `prefetchtrees`. Soon, we will merge
`prefetch` and `prefetchtrees` into a single command and both commands having a
similar interface is useful for the merging.
Test Plan: Ran all the tests.
Reviewers: #fbhgext, durham
Reviewed By: #fbhgext, durham
Subscribers: durham
Differential Revision: https://phab.mercurial-scm.org/D1368
The exec bit got lost after rebase.
The rebase bug was filed as https://bz.mercurial-scm.org/5743.
The file content change is to workaround a potential pushrebase bug that
does not allow mode-only change.
Previously the test sets up `LD_LIBRARY_PATH` and `PYTHONPATH`, then runs
Python tests.
Within Python code, setting `sys.path` would achieve the same effect of
setting `PYTHONPATH`. For `LD_LIBRARY_PATH`, it's necessary for C libraries.
But the only C library that cstore depends on is `lz4`, which is supposed to
use the system version. There is no C library provided by this repo -
features like sha1 are compiled in `cstore.so`.
Therefore it's unnecessary to have a separate `.t` file wrapping `.py`
tests. Let's just use `.py` tests directly.
Test Plan:
`./script/unit.py`
Make a temporary change to `cdatapack.c` so it fails unconditionally in
open_datapack. Build the repo in different ways: `make local` and
`python2 setup.py build_clib build_ext`. Then run the test by using
`$HG_CREW/tests/run-tests.py -l test-remotefilelog-datapack.py` without the
`hg-dev` environment and make sure it fails with the expected exception.
Differential Revision: https://phab.mercurial-scm.org/D1429
If the os limited a given process to a <200 files open, this test would fail.
Let's change the cache size to be smaller to avoid this.
Also, it turns out the cache size and number of packs created doesn't actually
seem to affect this test. I changed the numbers in a few ways and the test never
failed.
Differential Revision: https://phab.mercurial-scm.org/D1435
Add `repack.chainorphansbysize` (default True).
When enabled, we take all orphaned nodes (nodes that are not part of a chain),
and put them into a new chain at the end, so we can get some minimal
compression out of them. Right now, they default to each being stored as
fulltexts, which is wasteful.
We sort the orphan chain by size, descending, to make the largest version
quickest to access, on the assumption that it is probably the newest. (This is
what Git does for its packed data, and it is a decent fallback if ancestry is
not available)
Example chain output, before:
```
A->B C D->E->F G H
```
After:
```
A->B D->E->F G->C->H
(assuming len(G)>=C=>H)
```
(I'm still adding a test case, but the code itself could be reviewed.)
Differential Revision: https://phab.mercurial-scm.org/D1272
We got exception:
unable to load pack ...: [Errno 24] Too many open files
on OS X machine where we think `ulimit -n` is big enough.
Let's add some debugging outputs so we can have more clues about it.
Note: the Python implementation of `mmap.mmap` actually keeps a fd open [1].
So the fix (65c38ccb9835) only reduces fd count from 2 * N to N, but does
not really solve the issue.
We might want to enforce the native code path to work around Python mmap
implementation.
[1]: # Modules/mmapmodule.c
m_obj->fd = dup(fd);
if (m_obj->fd == -1) {
Py_DECREF(m_obj);
PyErr_SetFromErrno(mmap_module_error);
return NULL;
}
Differential Revision: https://phab.mercurial-scm.org/D1420
A very minor change, but we should probably explain that local rebasing is
needed. (You might be forgiven for thinking that `pushrebase` would have
done that for you.)
Differential Revision: https://phab.mercurial-scm.org/D1352
Print which bad characters were found on what line, so that users can
fix the problem just from the hook message.
Differential Revision: https://phab.mercurial-scm.org/D1419
Summary: The TODO has been addressed and this test should be able to run now.
Test Plan:
- Checked that test host is capable of running the test now.
- Ran all the tests.
Reviewers: #fbhgext, mitrandir
Reviewed By: #fbhgext, mitrandir
Differential Revision: https://phab.mercurial-scm.org/D1369
This lets you list your currently active profiles, as well as let you discover
new profiles, provided sparse.profile_directory is set.
Includes JSON output. Future revisions can build on this to provide richer
metadata (parsed from the profile files).
Differential Revision: https://phab.mercurial-scm.org/D1250
Occasionally, callers to `hg repack` prefer to skip loose objects and only
repack packfiles. This adds an option to do so.
Differential Revision: https://phab.mercurial-scm.org/D1228
Currently an incremental repack on the server will repack the entire pack files,
and the new parts of the revlogs. The pack files can be very large and can take
a long, long time to run. So let's use the normal incremental pack heuristics to
minimize how often we have to do full repacks.
Differential Revision: https://phab.mercurial-scm.org/D1350
Summary:
It's failing on our tests macs because they can't have that many files open at
the time
Test Plan: tested on my laptop, fingers crossed
Reviewers: #mercurial, ikostia
Reviewed By: ikostia
Subscribers: mjpieters, medson
Differential Revision: https://phabricator.intern.facebook.com/D6285344
Tasks: T23454758
Signature: 6285344:1510247301:f295431e05836921288c313034864c3ec616b8af
Summary:
To speed up pack lookups (especially when there are lots of packs), we
should maintain an lru ordering of the packs and perform searches in that
order, since it's likely the next entry we search for will be in the same pack
file as the last entry we searched for. This commit achieves the same.
Test Plan:
- Ran all the tests.
- Created ~2k pack files in a large repo.
- Time taken without the cache:
- `hg update b` while at a: ~18 minutes.
- `hg update a` while at b: ~23 seconds.
- Time taken with the cache:
- `hg update b` while at a: ~14 seconds.
- `hg update a` while at b: ~9 seconds.
Reviewers: #fbhgext, durham
Reviewed By: #fbhgext, durham
Subscribers: durham
Differential Revision: https://phab.mercurial-scm.org/D1208
Summary:
createPack had no option to specify the pack directory because of
which it can only create one pack in a directory. This restriction was in place
because we only test the datapack and not the datapackstore during these tests.
This commit makes the method more generic and includes the option to specify
the directory for creating the packs. This would allow for the datapackstore to
be tested while reusing most of the current logic.
Test Plan: Ran all the tests.
Reviewers: #fbhgext, durham
Reviewed By: #fbhgext, durham
Subscribers: durham
Differential Revision: https://phab.mercurial-scm.org/D1325
Summary:
Add support to hg book -d to delete scratch infinitepush bookmarks
Uses functions from remotenames to rewrite the remotenames cache omitting
the specified scratch bookmarks
Test Plan:
cd ~/facebook-hg-rpms/fb-hgext/tests
source ../../hg-dev
rt test-infinitepush-*.t --extra-config-opt=devel.all-warnings=False
Reviewers: #mercurial, cdown, stash, durham
Differential Revision: https://phabricator.intern.facebook.com/D6221853
Tasks: T22615396
There is a bug in the upstream bundlerepo implementation that causes it to
infinite loop if the manifestlog isn't backed by a revlog. I've sent a fix
upstream, and this adds a test to cover that case.
The core Mercurial manifest template prints the rev number and the short hash.
Since treemanifest doesn't have rev numbers, this has to change. Let's just have
it print the whole hash, since manifest hashes are usually only ever used by
automation which probably wants the whole hash anyway.
Differential Revision: https://phab.mercurial-scm.org/D1305
If a hybrid repo pulls in a treeonly commit from a treeonly client, it
previously couldn't commit on top of it because it tried to read the flat
manifest. This patch makes it possible for the hybrid repo to make a treeonly
commit if it is committing on top of a treeonly commit (i.e. where the manifest
only exists in the tree store, not in the flat manifest revlog).
This makes it easier for multiple types of repositories to interact, and to flip
back and forth between treeonly and non-treeonly as we migrate.
Differential Revision: https://phab.mercurial-scm.org/D1304
Also print the name of the packfile being inspected, and \itweak newlines a bit.
This simplies debugging several packfiles in tests.
Differential Revision: https://phab.mercurial-scm.org/D1326
When trees are fetched from the server as packs, metadata isn't included, as
it's not supported in the protocol. Fast size information is useful, since
we have access to the fulltext during a repack, add the metadatan then.
This will be needed for sized-based sorting of manifest entries.
Differential Revision: https://phab.mercurial-scm.org/D1255
This patch adds support to histedit command to take input from json. The JSON
should be of following format:
{ "histedit": [
{"action": "...", "node": "..."},
{"action": "...", "command": "..."},
{...}
]
}
For actions except "exec" and "execr", "node" is required and for "exec" and
"execr", "command" should be there.
The support is plugged in directly and is not hidden behind a config flag. After
this support, the parser which parses the commands file will assume that JSON is
given and will fallback to original parsing if it fails to convert the data in
the file to JSON.
In case where input is JSON but a key is missing, status message is shown and we
fallback to normal parsing.
Both small and large verbs are supported for the action part in JSON.
Differential Revision: https://phab.mercurial-scm.org/D1263
`.pyc` files could cause false positive. Scan `.py` and `.pyx` files
explicitly.
Test Plan:
Make sure `uncommit.pyc` will not break check-code but `uncommit.pyx`,
`uncommit.py` will.
Differential Revision: https://phab.mercurial-scm.org/D1283
With the multidest rebase (D470), restack could be implemented as specifying
the source (`orphan() - obsolete()`) and a revset specifying destination for
each source revision.
This patch changes restack implementation to use that. The revset is
implemented as a private function `_destrestack`, like `_destrebase` in
`rebase.py`.
Most test changes are because the topo-sort in `rebase.py` removes some
unnecessary steps. So the resulting revision numbers are smaller.
There is one interesting test case that gets changed:
```
D
| # amend: B1 -> B2 -> B1
B2 B1 B3 # amend: B1 -> B3
\ | /
A
```
Previously, restack will move D to be on top of B3. Now restack will not
move it because `D` is considered stable (not orphaned) since none of its
ancestors are obsoleted. The new behavior seems to be more desirable.
More tests are added to test restack source revisions (should be in a same
stack) and some "prune" cases.
The transaction logic is also made more robust so `test-copytrace-amend.t`
does not cause a broken rebase state.
Since the rebase operation runs in a same rebase state, restack can be
continued via `rebase --continue` correctly after resolving merge conflicts.
This is reflected in a newly added test.
Test Plan:
With hg-dev, run `./script/unit.py`
Differential Revision: https://phab.mercurial-scm.org/D1139
test-remotefilelog-clone.t is flaky, and fails sporadically as follows:
$ hg clone --noupdate ssh://user@dummy/shallow full
streaming all changes
+ abort: unexpected response from remote server: empty string
remote: abort: Cannot clone from a shallow repo to a full repo.
- abort: unexpected response from remote server: empty string
[255]
The issue happens because the line starting "abort" and the line starting
"remote: abort" come from different processes, and there's no way to
guarantee the order in which they are output.
This change attempts to address the issue by capturing the output of the
local process and examining it separately.
Test Plan:
rt test-remotefilelog-clone.t
Differential Revision: https://phab.mercurial-scm.org/D1279
Adds a test and improves output for dealing pulling a normal scratch branch into
a treeonly repo (so the manifest doesn't exist).
In the future we could fix this by having the server generate the trees on the
fly for infinitepush branches that don't already have them.
Differential Revision: https://phab.mercurial-scm.org/D1268
This makes the client side pushes also include tree packs in the push.
Also adds a test to see that the tree parts round trip between two tree-only
clients.
This was reverted in D921 because clients couldn't push old non-tree commits to
infinitepush anymore. I've fixed the issue and added a test to this commit.
Differential Revision: https://phab.mercurial-scm.org/D1017
The old version of this test relied on the sendflat config being set to false so
it would only send the trees. Since that config is gone, the test then started
sending flat manifests, so it wasn't actually testing what it should've been.
This patch fixes that by making it push a tree only commit and ensuring the
commit hook executes correctly. The test changed a bit, because the hash changed
after the push, because it was a tree only commit on the client (and therefore
used the tree hash) but a hybrid commit on the server (and therefore used the
flat hash).
Differential Revision: https://phab.mercurial-scm.org/D1267
Previously the sendflat config decided if we should send flat manifests or not.
To reduce the number of config permutations we have to handle, let's change the
code to always send the flat manifests, if it exists.
Differential Revision: https://phab.mercurial-scm.org/D1266
Previously we were relying on config values to determine when to send or not
send tree and flat manifests. This resulted in complicated permutations to test.
Let's remove these config values and always send trees if we have them. A future
patch will do the same for flat manifests.
Differential Revision: https://phab.mercurial-scm.org/D1265
Previously, hiddenoverride only unpins nodes when createmarkers gets called.
However, with scmutils.cleanupnodes, createmarkers will not get called when
the nodes are already obsoleted. This patch makes hiddenoverride check
cleanupnodes too to unpin nodes properly.
Differential Revision: https://phab.mercurial-scm.org/D994
Summary:
Use python instead of jq in pullbackup test
fixes breakage reported in D6186148
Test Plan:
Run unit tests
Verify build runs without a hitch
Reviewers: swhitaker
Differential Revision: https://phabricator.intern.facebook.com/D6208705
