Summary: The TODO has been addressed and this test should be able to run now.
Test Plan:
- Checked that test host is capable of running the test now.
- Ran all the tests.
Reviewers: #fbhgext, mitrandir
Reviewed By: #fbhgext, mitrandir
Differential Revision: https://phab.mercurial-scm.org/D1369
Treemanifest now has a unified spot to check if it can send trees. Infinitepush
needs to respect that, otherwise we're uploading trees to infinitepush that
might not be readable on other systems.
This is a common technique to store variable-length integers efficiently.
It's compatible with both Thrift and Protobuf [1].
It's intended to be used in:
- On-disk file format to make the file compact and avoid issues like
https://bz.mercurial-scm.org/5681 (Obsolete markers code crashes with
metadata keys/values longer than 255 bytes).
- Thrift layer.
[1]: https://developers.google.com/protocol-buffers/docs/encoding#varints
Test Plan:
```
cargo test
cargo clippy
```
Also ran a kcov coverage check and it says 100%.
```
cargo rustc --lib --profile test -- -Ccodegen-units=1 -Clink-dead-code -Zno-landing-pads
kcov --include-path $PWD/src --verify target/kcov ./target/debug/*-????????????????
```
Differential Revision: https://phab.mercurial-scm.org/D929
This lets you list your currently active profiles, as well as let you discover
new profiles, provided sparse.profile_directory is set.
Includes JSON output. Future revisions can build on this to provide richer
metadata (parsed from the profile files).
Differential Revision: https://phab.mercurial-scm.org/D1250
Occasionally, callers to `hg repack` prefer to skip loose objects and only
repack packfiles. This adds an option to do so.
Differential Revision: https://phab.mercurial-scm.org/D1228
Currently an incremental repack on the server will repack the entire pack files,
and the new parts of the revlogs. The pack files can be very large and can take
a long, long time to run. So let's use the normal incremental pack heuristics to
minimize how often we have to do full repacks.
Differential Revision: https://phab.mercurial-scm.org/D1350
This fixes blackbox.log to not have two messages on the same line. This might be
undesirable if there's some other system using ui.log and this was *expected* to
be creating a single line. In that case, this might instead be a feature request
for blackbox to not insert time/user/node/etc. if it's a consecutive log from
the same 'service'. Currently, the docstring for ui.log says "*msg should be a
newline-terminated format string to log", so this is bringing these uses in
line with that.
Sample blackbox.log without this fix:
2017/11/06 14:41:23 spectral @a659d684cdf40d442d38f1ea65ee618f8b21d4b6 (25545)> remote cache hit rate is 0 of 9 2017/11/06 14:41:23 spectral @a659d684cdf40d442d38f1ea65ee618f8b21d4b6 (25545)> Success2017/11/06 14:45:24 spectral @dcbd198c160cfc8fc6d4a877aa5ed9296f98ee3c (25545)> pythonhook-update: remotefilelog.wcpprefetch finished in 0.00 seconds
We need to pass the metadata store in, but we were passing the content store.
This only worked because currently we never use the metadata store on the
fileserver client for writing.
Summary:
It's failing on our tests macs because they can't have that many files open at
the time
Test Plan: tested on my laptop, fingers crossed
Reviewers: #mercurial, ikostia
Reviewed By: ikostia
Subscribers: mjpieters, medson
Differential Revision: https://phabricator.intern.facebook.com/D6285344
Tasks: T23454758
Signature: 6285344:1510247301:f295431e05836921288c313034864c3ec616b8af
Summary:
To speed up pack lookups (especially when there are lots of packs), we
should maintain an lru ordering of the packs and perform searches in that
order, since it's likely the next entry we search for will be in the same pack
file as the last entry we searched for. This commit achieves the same.
Test Plan:
- Ran all the tests.
- Created ~2k pack files in a large repo.
- Time taken without the cache:
- `hg update b` while at a: ~18 minutes.
- `hg update a` while at b: ~23 seconds.
- Time taken with the cache:
- `hg update b` while at a: ~14 seconds.
- `hg update a` while at b: ~9 seconds.
Reviewers: #fbhgext, durham
Reviewed By: #fbhgext, durham
Subscribers: durham
Differential Revision: https://phab.mercurial-scm.org/D1208
Summary:
createPack had no option to specify the pack directory because of
which it can only create one pack in a directory. This restriction was in place
because we only test the datapack and not the datapackstore during these tests.
This commit makes the method more generic and includes the option to specify
the directory for creating the packs. This would allow for the datapackstore to
be tested while reusing most of the current logic.
Test Plan: Ran all the tests.
Reviewers: #fbhgext, durham
Reviewed By: #fbhgext, durham
Subscribers: durham
Differential Revision: https://phab.mercurial-scm.org/D1325
Advice using `hg uncommit` when a command to prune (like `hg strip`) but keeping the changes (with `--keep` option)
Test Plan:
Run `hg strip -k\--keep` a "'hg uncommit' provides a better UI for undoing commits while keeping the changes" should show up.
Run `hg strip` w/o `--keep` option, "'hg hide' provides a better UI for hiding commits" should be shown
Differential Revision: https://phab.mercurial-scm.org/D1335
When the dirstate got refactored, we lost the check that only logged the
dirstate size if the dictionary was already populated. This caused a regression
in hg bookmark times (since it normally doesn't populate the dirstate map).
Summary:
Add support to hg book -d to delete scratch infinitepush bookmarks
Uses functions from remotenames to rewrite the remotenames cache omitting
the specified scratch bookmarks
Test Plan:
cd ~/facebook-hg-rpms/fb-hgext/tests
source ../../hg-dev
rt test-infinitepush-*.t --extra-config-opt=devel.all-warnings=False
Reviewers: #mercurial, cdown, stash, durham
Differential Revision: https://phabricator.intern.facebook.com/D6221853
Tasks: T22615396
An invalid entry is any entry with a base not in the pack, or whose deltabases
form a cycle.
If there are any entries like that, the output will look like this:
```
(Root):
Node Delta Base Delta Length Blob Size
665a7e7913af e66038a2894e 61 2142
52bd634be310 000000000000 2142 2142
8b5847087ce0 000000000000 2142 2142
960f5acb3e99 edf2ffd7daab 162 2142
b7d7e5aa692e 8b5847087ce0 162 2142
cdcc4d74d667 960f5acb3e99 324 2142
Total: 14652 48920 (70.0% smaller)
Bad entry: 960f5acb3e99 has an unknown deltabase (edf2ffd7daab)
Bad entry: b7d7e5aa692e has an unknown deltabase (edf2ffd7daab)
2 invalid entries
```
Differential Revision: https://phab.mercurial-scm.org/D1271
There is a bug in the upstream bundlerepo implementation that causes it to
infinite loop if the manifestlog isn't backed by a revlog. I've sent a fix
upstream, and this adds a test to cover that case.
The core Mercurial manifest template prints the rev number and the short hash.
Since treemanifest doesn't have rev numbers, this has to change. Let's just have
it print the whole hash, since manifest hashes are usually only ever used by
automation which probably wants the whole hash anyway.
Differential Revision: https://phab.mercurial-scm.org/D1305
If a hybrid repo pulls in a treeonly commit from a treeonly client, it
previously couldn't commit on top of it because it tried to read the flat
manifest. This patch makes it possible for the hybrid repo to make a treeonly
commit if it is committing on top of a treeonly commit (i.e. where the manifest
only exists in the tree store, not in the flat manifest revlog).
This makes it easier for multiple types of repositories to interact, and to flip
back and forth between treeonly and non-treeonly as we migrate.
Differential Revision: https://phab.mercurial-scm.org/D1304
When repacking data, we sort data nodes topologically by ancestry in order to
ensure the best (smallest) delta chain. Unfortunately the history we use to do
this will be whatever history packs the samre repack job chose for its history
repacking portion, which might be comically small and/or irrelevant.
To fix this, select all history packfiles, and pass them to the data packer as
`fullhistory`. Print a debug warning whenever any nodes are missing ancestry.
Differential Revision: https://phab.mercurial-scm.org/D1227
Also print the name of the packfile being inspected, and \itweak newlines a bit.
This simplies debugging several packfiles in tests.
Differential Revision: https://phab.mercurial-scm.org/D1326
The previous solution was incomplete. This solution logs once per run, with two
separate metrics (filestore_ and treestore_), each logging the number of packs
and bytes. I also did some refactoring.
Differential Revision: https://phab.mercurial-scm.org/D1309
When trees are fetched from the server as packs, metadata isn't included, as
it's not supported in the protocol. Fast size information is useful, since
we have access to the fulltext during a repack, add the metadatan then.
This will be needed for sized-based sorting of manifest entries.
Differential Revision: https://phab.mercurial-scm.org/D1255
Adds information that is useful for debugging infinitepush backup to the rage
output. In particular:
- Full node hashes in the unfiltered smartlog, so that they can be looked up
in infinitepush.
- The contents of infinitepushbackupstate.
- The last 100 lines of the infinitepush backup logs for this repo.
Differential Revision: https://phab.mercurial-scm.org/D1264
I wasn't 100% sure this is side effect-free if you enable the extension but
don't specify a progressfile, so let's be extra safe.
Differential Revision: https://phab.mercurial-scm.org/D1308
This patch adds support to histedit command to take input from json. The JSON
should be of following format:
{ "histedit": [
{"action": "...", "node": "..."},
{"action": "...", "command": "..."},
{...}
]
}
For actions except "exec" and "execr", "node" is required and for "exec" and
"execr", "command" should be there.
The support is plugged in directly and is not hidden behind a config flag. After
this support, the parser which parses the commands file will assume that JSON is
given and will fallback to original parsing if it fails to convert the data in
the file to JSON.
In case where input is JSON but a key is missing, status message is shown and we
fallback to normal parsing.
Both small and large verbs are supported for the action part in JSON.
Differential Revision: https://phab.mercurial-scm.org/D1263
It looks like the devel warning detection complains about reading
extensions.fbsparse with a default value. Let's just suppress it.
Differential Revision: https://phab.mercurial-scm.org/D1294
Summary:
Since reponame is part of the primary key, it also makes sense to add it to
WHERE clause. Otherwise query may trigger full table scan and it can be very
slow.
Test Plan: Run unittest
Reviewers: #fbhgext, durham
Reviewed By: #fbhgext, durham
Differential Revision: https://phab.mercurial-scm.org/D1256
`.pyc` files could cause false positive. Scan `.py` and `.pyx` files
explicitly.
Test Plan:
Make sure `uncommit.pyc` will not break check-code but `uncommit.pyx`,
`uncommit.py` will.
Differential Revision: https://phab.mercurial-scm.org/D1283
With the multidest rebase (D470), restack could be implemented as specifying
the source (`orphan() - obsolete()`) and a revset specifying destination for
each source revision.
This patch changes restack implementation to use that. The revset is
implemented as a private function `_destrestack`, like `_destrebase` in
`rebase.py`.
Most test changes are because the topo-sort in `rebase.py` removes some
unnecessary steps. So the resulting revision numbers are smaller.
There is one interesting test case that gets changed:
```
D
| # amend: B1 -> B2 -> B1
B2 B1 B3 # amend: B1 -> B3
\ | /
A
```
Previously, restack will move D to be on top of B3. Now restack will not
move it because `D` is considered stable (not orphaned) since none of its
ancestors are obsoleted. The new behavior seems to be more desirable.
More tests are added to test restack source revisions (should be in a same
stack) and some "prune" cases.
The transaction logic is also made more robust so `test-copytrace-amend.t`
does not cause a broken rebase state.
Since the rebase operation runs in a same rebase state, restack can be
continued via `rebase --continue` correctly after resolving merge conflicts.
This is reflected in a newly added test.
Test Plan:
With hg-dev, run `./script/unit.py`
Differential Revision: https://phab.mercurial-scm.org/D1139
test-remotefilelog-clone.t is flaky, and fails sporadically as follows:
$ hg clone --noupdate ssh://user@dummy/shallow full
streaming all changes
+ abort: unexpected response from remote server: empty string
remote: abort: Cannot clone from a shallow repo to a full repo.
- abort: unexpected response from remote server: empty string
[255]
The issue happens because the line starting "abort" and the line starting
"remote: abort" come from different processes, and there's no way to
guarantee the order in which they are output.
This change attempts to address the issue by capturing the output of the
local process and examining it separately.
Test Plan:
rt test-remotefilelog-clone.t
Differential Revision: https://phab.mercurial-scm.org/D1279
If we were using fastmanifest in a treeonly repository, hg show would try to
access the manifest delta via the revlog which would crash. The fix is to make
fastmanifest first try accessing the data via the tree, before falling back to
the delta.
My attempts at automatically testing this failed, but I had a repro in a real
repository and confirmed this fixed the issue.
There was a flakey test because the remote stderr output was getting eaten part
of the time. This fixes it by capturing all the output and then sending it to
stderr manually.
Tested it by running all the tests in parallel multiple times and not seeing the
issue anymore.
Also we now send that output to the debug output. We're already throwing a more
useful error message, so let's hide the noise.
Differential Revision: https://phab.mercurial-scm.org/D1276
Adds a test and improves output for dealing pulling a normal scratch branch into
a treeonly repo (so the manifest doesn't exist).
In the future we could fix this by having the server generate the trees on the
fly for infinitepush branches that don't already have them.
Differential Revision: https://phab.mercurial-scm.org/D1268
This makes the client side pushes also include tree packs in the push.
Also adds a test to see that the tree parts round trip between two tree-only
clients.
This was reverted in D921 because clients couldn't push old non-tree commits to
infinitepush anymore. I've fixed the issue and added a test to this commit.
Differential Revision: https://phab.mercurial-scm.org/D1017
The old version of this test relied on the sendflat config being set to false so
it would only send the trees. Since that config is gone, the test then started
sending flat manifests, so it wasn't actually testing what it should've been.
This patch fixes that by making it push a tree only commit and ensuring the
commit hook executes correctly. The test changed a bit, because the hash changed
after the push, because it was a tree only commit on the client (and therefore
used the tree hash) but a hybrid commit on the server (and therefore used the
flat hash).
Differential Revision: https://phab.mercurial-scm.org/D1267
