Previously the test sets up `LD_LIBRARY_PATH` and `PYTHONPATH`, then runs
Python tests.
Within Python code, setting `sys.path` would achieve the same effect of
setting `PYTHONPATH`. For `LD_LIBRARY_PATH`, it's necessary for C libraries.
But the only C library that cstore depends on is `lz4`, which is supposed to
use the system version. There is no C library provided by this repo -
features like sha1 are compiled in `cstore.so`.
Therefore it's unnecessary to have a separate `.t` file wrapping `.py`
tests. Let's just use `.py` tests directly.
Test Plan:
`./script/unit.py`
Make a temporary change to `cdatapack.c` so it fails unconditionally in
open_datapack. Build the repo in different ways: `make local` and
`python2 setup.py build_clib build_ext`. Then run the test by using
`$HG_CREW/tests/run-tests.py -l test-remotefilelog-datapack.py` without the
`hg-dev` environment and make sure it fails with the expected exception.
Differential Revision: https://phab.mercurial-scm.org/D1429
If the os limited a given process to a <200 files open, this test would fail.
Let's change the cache size to be smaller to avoid this.
Also, it turns out the cache size and number of packs created doesn't actually
seem to affect this test. I changed the numbers in a few ways and the test never
failed.
Differential Revision: https://phab.mercurial-scm.org/D1435
Added workers in lfs.
I had to remove the fine progress tracking because between processes in *nix and threads in windows (diffs will appear soon) the tracking of 1MB progress is quite tricky.
With our network tracking progress per file is way enough to see things moving.
This change gives close to 50% speedup on hg sparse --enable-profile when prefetch is run. My current understanding is that prefetch is ran when profile is enabled for the first time.
Test Plan:
Enable profile:
time hg sprase --enable-profile SparseProfiles/TestProfile.sparse
The profile contains 42k files including 9GB of lfs files
On my machine the time improves by 47% while still being dominated by lfs
download time
# Tip: In Git and Mercurial, use a branch like "T123" to automatically associate
# changes with the corresponding task.
Differential Revision: https://phab.mercurial-scm.org/D1424
Add `repack.chainorphansbysize` (default True).
When enabled, we take all orphaned nodes (nodes that are not part of a chain),
and put them into a new chain at the end, so we can get some minimal
compression out of them. Right now, they default to each being stored as
fulltexts, which is wasteful.
We sort the orphan chain by size, descending, to make the largest version
quickest to access, on the assumption that it is probably the newest. (This is
what Git does for its packed data, and it is a decent fallback if ancestry is
not available)
Example chain output, before:
```
A->B C D->E->F G H
```
After:
```
A->B D->E->F G->C->H
(assuming len(G)>=C=>H)
```
(I'm still adding a test case, but the code itself could be reviewed.)
Differential Revision: https://phab.mercurial-scm.org/D1272
We got exception:
unable to load pack ...: [Errno 24] Too many open files
on OS X machine where we think `ulimit -n` is big enough.
Let's add some debugging outputs so we can have more clues about it.
Note: the Python implementation of `mmap.mmap` actually keeps a fd open [1].
So the fix (65c38ccb9835) only reduces fd count from 2 * N to N, but does
not really solve the issue.
We might want to enforce the native code path to work around Python mmap
implementation.
[1]: # Modules/mmapmodule.c
m_obj->fd = dup(fd);
if (m_obj->fd == -1) {
Py_DECREF(m_obj);
PyErr_SetFromErrno(mmap_module_error);
return NULL;
}
Differential Revision: https://phab.mercurial-scm.org/D1420
A very minor change, but we should probably explain that local rebasing is
needed. (You might be forgiven for thinking that `pushrebase` would have
done that for you.)
Differential Revision: https://phab.mercurial-scm.org/D1352
Print which bad characters were found on what line, so that users can
fix the problem just from the hook message.
Differential Revision: https://phab.mercurial-scm.org/D1419
When we changed the treemanifest {manifest} template output it broke the ability
to specifically ask for the node. This is important for tools migrating between
the old and new format. Let's add that back in.
Let's also make tweakdefaults change the '{manifest}' default template for all
repo's, not just tree repos.
Differential Revision: https://phab.mercurial-scm.org/D1418
Summary: The TODO has been addressed and this test should be able to run now.
Test Plan:
- Checked that test host is capable of running the test now.
- Ran all the tests.
Reviewers: #fbhgext, mitrandir
Reviewed By: #fbhgext, mitrandir
Differential Revision: https://phab.mercurial-scm.org/D1369
Treemanifest now has a unified spot to check if it can send trees. Infinitepush
needs to respect that, otherwise we're uploading trees to infinitepush that
might not be readable on other systems.
This is a common technique to store variable-length integers efficiently.
It's compatible with both Thrift and Protobuf [1].
It's intended to be used in:
- On-disk file format to make the file compact and avoid issues like
https://bz.mercurial-scm.org/5681 (Obsolete markers code crashes with
metadata keys/values longer than 255 bytes).
- Thrift layer.
[1]: https://developers.google.com/protocol-buffers/docs/encoding#varints
Test Plan:
```
cargo test
cargo clippy
```
Also ran a kcov coverage check and it says 100%.
```
cargo rustc --lib --profile test -- -Ccodegen-units=1 -Clink-dead-code -Zno-landing-pads
kcov --include-path $PWD/src --verify target/kcov ./target/debug/*-????????????????
```
Differential Revision: https://phab.mercurial-scm.org/D929
This lets you list your currently active profiles, as well as let you discover
new profiles, provided sparse.profile_directory is set.
Includes JSON output. Future revisions can build on this to provide richer
metadata (parsed from the profile files).
Differential Revision: https://phab.mercurial-scm.org/D1250
Occasionally, callers to `hg repack` prefer to skip loose objects and only
repack packfiles. This adds an option to do so.
Differential Revision: https://phab.mercurial-scm.org/D1228
Currently an incremental repack on the server will repack the entire pack files,
and the new parts of the revlogs. The pack files can be very large and can take
a long, long time to run. So let's use the normal incremental pack heuristics to
minimize how often we have to do full repacks.
Differential Revision: https://phab.mercurial-scm.org/D1350
This fixes blackbox.log to not have two messages on the same line. This might be
undesirable if there's some other system using ui.log and this was *expected* to
be creating a single line. In that case, this might instead be a feature request
for blackbox to not insert time/user/node/etc. if it's a consecutive log from
the same 'service'. Currently, the docstring for ui.log says "*msg should be a
newline-terminated format string to log", so this is bringing these uses in
line with that.
Sample blackbox.log without this fix:
2017/11/06 14:41:23 spectral @a659d684cdf40d442d38f1ea65ee618f8b21d4b6 (25545)> remote cache hit rate is 0 of 9 2017/11/06 14:41:23 spectral @a659d684cdf40d442d38f1ea65ee618f8b21d4b6 (25545)> Success2017/11/06 14:45:24 spectral @dcbd198c160cfc8fc6d4a877aa5ed9296f98ee3c (25545)> pythonhook-update: remotefilelog.wcpprefetch finished in 0.00 seconds
We need to pass the metadata store in, but we were passing the content store.
This only worked because currently we never use the metadata store on the
fileserver client for writing.
Summary:
It's failing on our tests macs because they can't have that many files open at
the time
Test Plan: tested on my laptop, fingers crossed
Reviewers: #mercurial, ikostia
Reviewed By: ikostia
Subscribers: mjpieters, medson
Differential Revision: https://phabricator.intern.facebook.com/D6285344
Tasks: T23454758
Signature: 6285344:1510247301:f295431e05836921288c313034864c3ec616b8af
Summary:
To speed up pack lookups (especially when there are lots of packs), we
should maintain an lru ordering of the packs and perform searches in that
order, since it's likely the next entry we search for will be in the same pack
file as the last entry we searched for. This commit achieves the same.
Test Plan:
- Ran all the tests.
- Created ~2k pack files in a large repo.
- Time taken without the cache:
- `hg update b` while at a: ~18 minutes.
- `hg update a` while at b: ~23 seconds.
- Time taken with the cache:
- `hg update b` while at a: ~14 seconds.
- `hg update a` while at b: ~9 seconds.
Reviewers: #fbhgext, durham
Reviewed By: #fbhgext, durham
Subscribers: durham
Differential Revision: https://phab.mercurial-scm.org/D1208
Summary:
createPack had no option to specify the pack directory because of
which it can only create one pack in a directory. This restriction was in place
because we only test the datapack and not the datapackstore during these tests.
This commit makes the method more generic and includes the option to specify
the directory for creating the packs. This would allow for the datapackstore to
be tested while reusing most of the current logic.
Test Plan: Ran all the tests.
Reviewers: #fbhgext, durham
Reviewed By: #fbhgext, durham
Subscribers: durham
Differential Revision: https://phab.mercurial-scm.org/D1325
Advice using `hg uncommit` when a command to prune (like `hg strip`) but keeping the changes (with `--keep` option)
Test Plan:
Run `hg strip -k\--keep` a "'hg uncommit' provides a better UI for undoing commits while keeping the changes" should show up.
Run `hg strip` w/o `--keep` option, "'hg hide' provides a better UI for hiding commits" should be shown
Differential Revision: https://phab.mercurial-scm.org/D1335
When the dirstate got refactored, we lost the check that only logged the
dirstate size if the dictionary was already populated. This caused a regression
in hg bookmark times (since it normally doesn't populate the dirstate map).
Summary:
Add support to hg book -d to delete scratch infinitepush bookmarks
Uses functions from remotenames to rewrite the remotenames cache omitting
the specified scratch bookmarks
Test Plan:
cd ~/facebook-hg-rpms/fb-hgext/tests
source ../../hg-dev
rt test-infinitepush-*.t --extra-config-opt=devel.all-warnings=False
Reviewers: #mercurial, cdown, stash, durham
Differential Revision: https://phabricator.intern.facebook.com/D6221853
Tasks: T22615396
An invalid entry is any entry with a base not in the pack, or whose deltabases
form a cycle.
If there are any entries like that, the output will look like this:
```
(Root):
Node Delta Base Delta Length Blob Size
665a7e7913af e66038a2894e 61 2142
52bd634be310 000000000000 2142 2142
8b5847087ce0 000000000000 2142 2142
960f5acb3e99 edf2ffd7daab 162 2142
b7d7e5aa692e 8b5847087ce0 162 2142
cdcc4d74d667 960f5acb3e99 324 2142
Total: 14652 48920 (70.0% smaller)
Bad entry: 960f5acb3e99 has an unknown deltabase (edf2ffd7daab)
Bad entry: b7d7e5aa692e has an unknown deltabase (edf2ffd7daab)
2 invalid entries
```
Differential Revision: https://phab.mercurial-scm.org/D1271
There is a bug in the upstream bundlerepo implementation that causes it to
infinite loop if the manifestlog isn't backed by a revlog. I've sent a fix
upstream, and this adds a test to cover that case.
The core Mercurial manifest template prints the rev number and the short hash.
Since treemanifest doesn't have rev numbers, this has to change. Let's just have
it print the whole hash, since manifest hashes are usually only ever used by
automation which probably wants the whole hash anyway.
Differential Revision: https://phab.mercurial-scm.org/D1305
If a hybrid repo pulls in a treeonly commit from a treeonly client, it
previously couldn't commit on top of it because it tried to read the flat
manifest. This patch makes it possible for the hybrid repo to make a treeonly
commit if it is committing on top of a treeonly commit (i.e. where the manifest
only exists in the tree store, not in the flat manifest revlog).
This makes it easier for multiple types of repositories to interact, and to flip
back and forth between treeonly and non-treeonly as we migrate.
Differential Revision: https://phab.mercurial-scm.org/D1304
When repacking data, we sort data nodes topologically by ancestry in order to
ensure the best (smallest) delta chain. Unfortunately the history we use to do
this will be whatever history packs the samre repack job chose for its history
repacking portion, which might be comically small and/or irrelevant.
To fix this, select all history packfiles, and pass them to the data packer as
`fullhistory`. Print a debug warning whenever any nodes are missing ancestry.
Differential Revision: https://phab.mercurial-scm.org/D1227
Also print the name of the packfile being inspected, and \itweak newlines a bit.
This simplies debugging several packfiles in tests.
Differential Revision: https://phab.mercurial-scm.org/D1326
The previous solution was incomplete. This solution logs once per run, with two
separate metrics (filestore_ and treestore_), each logging the number of packs
and bytes. I also did some refactoring.
Differential Revision: https://phab.mercurial-scm.org/D1309
When trees are fetched from the server as packs, metadata isn't included, as
it's not supported in the protocol. Fast size information is useful, since
we have access to the fulltext during a repack, add the metadatan then.
This will be needed for sized-based sorting of manifest entries.
Differential Revision: https://phab.mercurial-scm.org/D1255
Adds information that is useful for debugging infinitepush backup to the rage
output. In particular:
- Full node hashes in the unfiltered smartlog, so that they can be looked up
in infinitepush.
- The contents of infinitepushbackupstate.
- The last 100 lines of the infinitepush backup logs for this repo.
Differential Revision: https://phab.mercurial-scm.org/D1264
I wasn't 100% sure this is side effect-free if you enable the extension but
don't specify a progressfile, so let's be extra safe.
Differential Revision: https://phab.mercurial-scm.org/D1308
This patch adds support to histedit command to take input from json. The JSON
should be of following format:
{ "histedit": [
{"action": "...", "node": "..."},
{"action": "...", "command": "..."},
{...}
]
}
For actions except "exec" and "execr", "node" is required and for "exec" and
"execr", "command" should be there.
The support is plugged in directly and is not hidden behind a config flag. After
this support, the parser which parses the commands file will assume that JSON is
given and will fallback to original parsing if it fails to convert the data in
the file to JSON.
In case where input is JSON but a key is missing, status message is shown and we
fallback to normal parsing.
Both small and large verbs are supported for the action part in JSON.
Differential Revision: https://phab.mercurial-scm.org/D1263
It looks like the devel warning detection complains about reading
extensions.fbsparse with a default value. Let's just suppress it.
Differential Revision: https://phab.mercurial-scm.org/D1294
