Summary: Build the new Rust matcher with both buck and setup.py
Reviewed By: markbt
Differential Revision: D7319607
fbshipit-source-id: c5944a28602495a9127acb20b59eb95632a9a1f5
Summary:
It only contains a `gitignorematcher` which exposes `GitignoreMatcher`
features to Python.
Reviewed By: markbt
Differential Revision: D7319605
fbshipit-source-id: 846964a551813f9b0933bc30f4a0ba3f85362944
Summary:
The "pathmatcher" crate is intended to eventually cover more "matcher"
abilities so all Python "matcher" related logic can be handled by Rust.
For now, it only contains a gitignore matcher.
The gitignore matcher is designed to work in a repo (no need to create
multiple gitignore matchers for a repo from a higher layer), and be lazy
i.e. be tree-aware, and do not parse ".gitignore" unless necessary.
Worth mentioning that the gitignore logic provided by the "ignore" crate
seems decent in time complexity - it uses regular expression, which uses state
machines to achieve "testing against multiple patterns at once", instead of
testing patterns one-by-one like what git currently does.
Note: The "ignore" crate provides a nice "Walker" interface but that does
not fit very well with the required laziness here. So the walker interface
is not used.
Reviewed By: markbt
Differential Revision: D7319609
fbshipit-source-id: ebd131adf45a38f83acdf653f5e49d0624012152
Summary:
- Support 8 color mode, since some terminals (emacs) do not support 16
colors (i.e. "bright*" colors have no effects).
- Detect emacs and use 8 color mode automatically.
- Read terminfo colors. Respect it if it says the terminal supports more
colors. Suggested by wez.
- Read HGCOLORS. Respect it unconditionally. Change run-tests.py to set it.
- Change diff related colors to also fallback to 8 colors since "bright"
colors are not guaranteed available.
Differential Revision: D7432965
fbshipit-source-id: da75829f856b4de737b946af72d24ff5351026cb
Summary:
Adds two new template functions:
* `simpledate` formats recent dates in a human-friendly way, for example `Today
at 12:00`, `Wednesday at 07:30`, or `Jan 05 at 23:45`. The timezone used
can be overridden.
* `smartdate` shows different results depending on whether a date is recent
or not, where "recent" is defined by a threshold parameter.
Reviewed By: quark-zju
Differential Revision: D7123734
fbshipit-source-id: 7b68207e4debc85c0bfa72bba9b375a4aa11d7c6
Summary: This makes it clear the code has different code paths for on-disk entries.
Reviewed By: DurhamG
Differential Revision: D7422836
fbshipit-source-id: 018fa0e2c20682d4e1beba99f3307550e1f40388
Summary:
Add benchmarks inserting / looking up 20K entries.
Benchmark results on my laptop are:
index insertion time: [6.5339 ms 6.8174 ms 7.1805 ms]
index flush time: [15.651 ms 16.103 ms 16.537 ms]
index lookup (memory) time: [3.6995 ms 4.0252 ms 4.3046 ms]
index lookup (disk) time: [1.9986 ms 2.1224 ms 2.2464 ms]
index clone (memory) time: [2.5943 ms 2.6866 ms 2.7749 ms]
index clone (disk) time: [5.2302 us 5.5477 us 5.9518 us]
Comparing with highly optimized radixbuf:
index insertion time: [991.89 us 1.1708 ms 1.3844 ms]
index lookup time: [863.83 us 945.69 us 1.0304 ms]
Insertion takes 6x time. Lookup from memory takes 1.4x time, from disk takes
2.2x time. Flushing is the slowest - it needs 16x radixbuf insertion time.
Note: need to subtract "clone" time from "lookup" to get meaningful values
about "lookup". This cannot be done automatically due to the limitation of the
benchmark framework.
Although it's slower than radixbuf, the index is still faster than gdbm and
rocksdb. Note: the index does less than gdbm/rocksdb since it does not return
a `[u8]`-ish which requires extra lookups. So it's not a very fair comparison.
gdbm insertion time: [69.607 ms 75.102 ms 79.334 ms]
gdbm lookup time: [9.0855 ms 9.8480 ms 10.637 ms]
gdbm prepare time: [110.35 us 120.40 us 135.63 us]
rocksdb insertion time: [117.96 ms 123.42 ms 127.85 ms]
rocksdb lookup time: [24.413 ms 26.147 ms 28.153 ms]
rocksdb prepare time: [3.8316 ms 4.1776 ms 4.5039 ms]
Note: Subtract "prepare" from "insertion" to get meaningful values.
Code to benchmark rocksdb and gdbm:
```
extern crate criterion;
extern crate gnudbm;
extern crate rand;
extern crate rocksdb;
extern crate tempdir;
use criterion::Criterion;
use gnudbm::GdbmOpener;
use rand::{ChaChaRng, Rng};
use rocksdb::DB;
use tempdir::TempDir;
const N: usize = 20480;
/// Generate random buffer
fn gen_buf(size: usize) -> Vec<u8> {
let mut buf = vec![0u8; size];
ChaChaRng::new_unseeded().fill_bytes(buf.as_mut());
buf
}
fn criterion_benchmark(c: &mut Criterion) {
c.bench_function("rocksdb prepare", |b| {
b.iter(move || {
let dir = TempDir::new("index").expect("TempDir::new");
let _db = DB::open_default(dir.path().join("a")).unwrap();
});
});
c.bench_function("rocksdb insertion", |b| {
let buf = gen_buf(N * 20);
b.iter(move || {
let dir = TempDir::new("index").expect("TempDir::new");
let db = DB::open_default(dir.path().join("a")).unwrap();
for i in 0..N {
db.put(&&buf[20 * i..20 * (i + 1)], b"v").unwrap();
}
});
});
c.bench_function("rocksdb lookup", |b| {
let dir = TempDir::new("index").expect("TempDir::new");
let db = DB::open_default(dir.path().join("a")).unwrap();
let buf = gen_buf(N * 20);
for i in 0..N {
db.put(&&buf[20 * i..20 * (i + 1)], b"v").unwrap();
}
b.iter(move || {
for i in 0..N {
db.get(&&buf[20 * i..20 * (i + 1)]).unwrap();
}
});
});
c.bench_function("gdbm prepare", |b| {
let buf = gen_buf(N * 20);
b.iter(move || {
let dir = TempDir::new("index").expect("TempDir::new");
let _db = GdbmOpener::new().create(true).readwrite(dir.path().join("a")).unwrap();
});
});
c.bench_function("gdbm insertion", |b| {
let buf = gen_buf(N * 20);
b.iter(move || {
let dir = TempDir::new("index").expect("TempDir::new");
let mut db = GdbmOpener::new().create(true).readwrite(dir.path().join("a")).unwrap();
for i in 0..N {
db.store(&&buf[20 * i..20 * (i + 1)], b"v").unwrap();
}
});
});
c.bench_function("gdbm lookup", |b| {
let dir = TempDir::new("index").expect("TempDir::new");
let mut db = GdbmOpener::new().create(true).readwrite(dir.path().join("a")).unwrap();
let buf = gen_buf(N * 20);
for i in 0..N {
db.store(&&buf[20 * i..20 * (i + 1)], b"v").unwrap();
}
b.iter(move || {
for i in 0..N {
db.fetch(&&buf[20 * i..20 * (i + 1)]).unwrap();
}
});
});
}
criterion_group!{
name=benches;
config=Criterion::default().sample_size(20);
targets=criterion_benchmark
}
criterion_main!(benches);
```
Reviewed By: DurhamG
Differential Revision: D7404532
fbshipit-source-id: ff39f520b78ad1b71eb36970506b313bb2ff426b
Summary:
This will be useful for benchmarks - prepare an index as a template, and
clone it in the tests.
Reviewed By: DurhamG
Differential Revision: D7422835
fbshipit-source-id: 190bbdee7cb7c1526274b4d4dab07af4984b5df6
Summary:
The latest rustfmt disagrees about the order of `std::io` imports. Move the
troublesome line to a separate group so both the old and new rustfmt agress
on the format.
Reviewed By: DurhamG
Differential Revision: D7422834
fbshipit-source-id: 9f5289ef2af1a691559fe691e121190f6d845162
Summary:
The old `rustc-test` crate no longer works. There is an upstream
bug report at https://github.com/servo/rustc-test/issues/7.
This change makes it possible to compare radixbuf performance
with the new index.
Reviewed By: DurhamG
Differential Revision: D7404531
fbshipit-source-id: 515e732a65388db4c865c7b139d0f57ead76f788
Summary:
Radix entries need to be written in an reversed order given the order they
are added to the vector.
Reviewed By: DurhamG
Differential Revision: D7404530
fbshipit-source-id: 403189b5c0fa6f21183e62eea04ce4ce7c4e1129
Summary: Those little read and write helpers are used in the next diff.
Reviewed By: DurhamG
Differential Revision: D7377214
fbshipit-source-id: c6e2d240334c11a0b08b15cd7d5c114b6f4d8ace
Summary:
Add a helper function `peek_key_entry_content` that checks key type and
return the key content.
Reviewed By: DurhamG
Differential Revision: D7377211
fbshipit-source-id: 0ce509aba30309373a709cf5fbcb909dd80471dc
Summary:
Implement insertion when there is no need to split a leaf entry.
The API may be subject to change if we want other value types. For now, it's
better to get something working and can be benchmarked so we have data about
performance impact with new format changes.
Reviewed By: DurhamG
Differential Revision: D7343423
fbshipit-source-id: 9761f72168046dbafcb00883634aa7ad513a522b
Summary:
Like the `peek_` family of helper methods. Those methods handles writing
data for both dirty (in-memory) and non-dirty (on-disk) cases. They will
be used in the next diff.
Reviewed By: DurhamG
Differential Revision: D7377208
fbshipit-source-id: f458a20da4bb7808f37daeed3077be2f7e90a9df
Summary:
Add code to print out Index's on-disk and in-memory entries in
human-friendly form. This is useful for explaining its internal state, so it
could be used in tests.
Reviewed By: DurhamG
Differential Revision: D7343427
fbshipit-source-id: 706a35404ea42c413657b389166729f8dd1315a3
Summary:
Offset stored in it needs to be translated, as done in other types of
entries. I forgot it.
Reviewed By: DurhamG
Differential Revision: D7404528
fbshipit-source-id: fb09a9c3052ddfe8f8016440290062084d5d8b03
Summary:
This is a low-level API that follows the base16 sequence of a key, and
return potentially matched `LinkOffset`.
Reviewed By: DurhamG
Differential Revision: D7343424
fbshipit-source-id: 38f260064d1a23695a28dda6f7dc921f88c7fccc
Summary:
Add a bunch of helper methods to "peek" data inside all kinds of entries.
They will be used in the next diff.
The benefit of those helper methods is they handle both dirty offsets and
non-dirty offsets transparently. Previously I have tried to always parse
on-disk entries into in-memory ones and stored them in a hashmap cache.
But that turned to have too much overhead so always reading from disk is
more desirable. It seems to provide at least 2x perf improvement from my
previous quick test.
Reviewed By: DurhamG
Differential Revision: D7377207
fbshipit-source-id: 1b393f1fe64c1d54b986ba7c3b03c790adb694d4
Summary:
The `non_dirty` helper method enforces the offset to be a non-dirty one.
It will be used frequently for checking offsets read from the disk, since
the on-disk offsets shouldn't have any reference to dirty (in-memory)
entries.
Reviewed By: DurhamG
Differential Revision: D7377209
fbshipit-source-id: c6c381c065d3ba8aaa65698224e4778b86edbc4a
Summary:
Some profiles are not for general consumption; they are usually profiles aimed at CI subsystems or similar. These can be hidden from listings by default using a `hidden` key.
The value of the key doesn't matter but can be used to explain why it is hidden.
Reviewed By: quark-zju
Differential Revision: D7433781
fbshipit-source-id: 877cd8698d50dc64cec8da706ab005e1fd786de4
Summary:
The hg sparse help page was overwhelming, and out of date. With subcommands, we
can break up the information a bit more.
* Subcommand help is the place for details on that command. Reduce the main
help info to an overview.
* Move configuration switches to a `verbose` container to hide it by default.
* Add documentation on the sparse profile format.
* Mark the command-line switches with subcommands as deprecated so they are
hidden by default.
Reviewed By: quark-zju
Differential Revision: D7432862
fbshipit-source-id: ffddd27e0ee4216f7e743e63e8efaed4eeaac582
Summary: This command mostly echoes how `hg files` works, albeit simplified somewhat. Given a profile and optional patterns, the files that profile matches are listed.
Reviewed By: quark-zju
Differential Revision: D7431522
fbshipit-source-id: 8e9d1f0d2aa31335f53269f3005875a0cc2e65bf
Summary:
Show the 'impact' of a profile, relative to a non-sparse working copy.
By default, this is the percentage of the total file count; adding --verbose will show the file sizes too.
The defined matchers have been refactored to reuse more of mercurial.match.basematcher, making it easier to reuse these in a wider mercurial context and avoiding repetition of common methods.
Reviewed By: quark-zju
Differential Revision: D7415720
fbshipit-source-id: 4ac3492c61aa70ee71d4bdf8c201b905a345a9d1
Summary: On my system, the `outgoing()` revset could easily take several seconds. A spinner here helps dispell the notion that mercurial has hung.
Reviewed By: quark-zju
Differential Revision: D7429086
fbshipit-source-id: 28908a34798d985dad3120647c3a5f474ca8a746
Summary:
Previously we relied on the list of files created by strip to decide
what directories to strip. It turns out this list wasn't adequate though, since
it's possible for directories to change even when files do not. For example,
during a merge if different files were added on different sides of the merge,
the files don't change during the merge but their containing directory does.
This diff special cases merge commits to make sure all the directories affected
by the merge are included in the strip.
Reviewed By: quark-zju
Differential Revision: D7409156
fbshipit-source-id: 9bf67eefb70189300c29db60d9945a7f608dfdda
Summary:
Previously we were trying to send trees to all clients during an
infinitepull, even ones that didn't support treemanifest. This caused
infinitepulls that required rebundling to fail for non-tree clients.
The fix is to just not send them unless the client is advertising the
capability.
Reviewed By: phillco
Differential Revision: D7432374
fbshipit-source-id: 1fae14a158ef56fe39439a718b1b98928f4e07b0
Summary: The order of the two reported files can differ, depending on the random hash seed.
Reviewed By: ryanmce
Differential Revision: D7427725
fbshipit-source-id: 9ec76867877553a02d1a19642d98d626b0ba91cc
Summary:
I need to be able to test for the presence of a cache without the ability to use a function, and I rather not use a sentinel value or have func raise an exception.
Note that the extension already is vulnerable to race conditions (there is plenty of time between the get, the func() call, and the set as it is).
Also, make it trivial to use JSON as a serializer.
Reviewed By: quark-zju
Differential Revision: D7415721
fbshipit-source-id: dc06e8f3efe725858a7960f7acfecc5a60390b85
Summary:
This gives us access to the specifics of a subcommand, including switches
(still stubbed out). Note: `hg sparse <subcommand> -h` doesn't work yet, as the
`-h` command line switch would need intercepting as well.
Reviewed By: quark-zju
Differential Revision: D7413859
fbshipit-source-id: 02534f1255a11985ebe8636b26946fbeffc6fe09
Summary:
Add the ability to set a `formatfunc` on a progres bar, which formats the
numbers used. Use `util.bytecount` as a format function in the places
where we have progress bars in numbers of bytes.
Reviewed By: quark-zju
Differential Revision: D7355232
fbshipit-source-id: 117c7035d46d47259cdfd70b80438cc6f4615977
Summary:
Nothing currently displays a progress bar while the curses interface is shown,
but in the future something may be added which does, so ensure the progress bar
is suspended when showing the curses interface.
Reviewed By: ryanmce
Differential Revision: D7417435
fbshipit-source-id: 6b91b17ee5390cbde6e983081a0940051ab865c8
Summary:
When updating a working copy that is limited by a sparse profile, do not abort on case collisions outside of the current working copy view.
Performance-wise, this isn't a big change; previously `_checkcollision()` would convert the whole manifest to a set, now we filter first with the matcher, then convert the remainder to a set. Either the original set conversion or the filtering can take O(size-of-manifest) time.
Reviewed By: ryanmce
Differential Revision: D7350251
fbshipit-source-id: efb8e270631f0aaa75c34c7b68189c60efe45984
Summary: progress.bar() is incorrectly called without passing ui.
Differential Revision: D7415250
fbshipit-source-id: 22c7419561879ed9293e2c79cc9d4271e805be76
Summary:
fix bug with new shelved() revsetpredicate
it failed if old format shelve files were present
Differential Revision: D7414883
fbshipit-source-id: 7b96abe7abf950ba339c22d0e9025f7423004911
Summary:
D7397947 broke `test-check-code.t` which complains about the use of
alias instead of function. This is blocking our continuous builds. This commit
addresses the same.
Reviewed By: quark-zju
Differential Revision: D7404361
fbshipit-source-id: 7907e913327c77861258a37ff8b87d467a7a15c6
Summary: This script can be used to generate a large number of commits.
Reviewed By: DurhamG
Differential Revision: D7363243
fbshipit-source-id: 1a1e3ba7fca29dc695446eeffb715474ea2943ad
Summary:
In uisetup we wrap a bunch of functions. We need to make sure they each
handle the case where treemanifest is not actually enabled for that repo. This
can happen in cases like a peer-to-peer pull where one repo has treemanifest
enabled and one does not.
Reviewed By: quark-zju
Differential Revision: D7368084
fbshipit-source-id: 152a448fb8a902e8c8f8ac4e806ab53514fadaa7
Summary: This shows basic details about a template, including base metadata. Once we have support for per-subcommand switches we can add further filtering.
Reviewed By: ryanmce
Differential Revision: D7365794
fbshipit-source-id: 3dd362f099e7f4d6db73c9bda7c24a9f4f2f90ee
Summary: Now all commands can be subcommands. The command-line switches will be hidden in a next commit (but remain available for backwards compatibility).
Reviewed By: ryanmce
Differential Revision: D7365393
fbshipit-source-id: 29b8ed2ac6bffccda41ed3f738087dde23312a8c
Summary:
This means we can
- count on the minirst module to format longer lines correctly,
- show subcommands when an error occurs (so full is false),
- add verbose help when the -v switch
Reviewed By: ryanmce
Differential Revision: D7365392
fbshipit-source-id: c8711bd19a6d992ad86f24f3c07b381f69d02121
Summary:
We take help info from the docstring but that'd disallow multiple registrations for a single function that can handle multiple subcommands. Refactor to let the decorator set an override.
So before you'd do this:
```
subcmd('foo')
def _foocmd(...):
"Help for foo command"
# implementation for foo
subcmd('bar')
def _barcmd(...):
"Help for bar command"
# implementation for bar
```
but you could not *combine* the implementations into one function and decorate both, because there is only the single docstring.
Now you can do:
```
subcmd('foo', help="Help for foo command")
subcmd('bar', help="Help for bar command")
def _fooandbarcmd(cmd, ...):
"This docstring is no longer used"
# implementation for foo and bar, combined, switching on the value of cmd
```
Reviewed By: ryanmce
Differential Revision: D7365390
fbshipit-source-id: 3d41542f1f197137ef13458e8d850cda8f53da74
Summary:
I want to add to helpcmd but with nested functions that's a lot more
complicated then it needs to be.
Reviewed By: ryanmce
Differential Revision: D7365391
fbshipit-source-id: 02092dd55f8f9521324b8ed51fa3134817454d36
Summary:
obsshelve: highlight shelved changes in smartlog (you can see with --hidden)
we highlighted only active shelved changes, those that has been unshelved are just ordinary hidden changes.
Reviewed By: ryanmce
Differential Revision: D7383800
fbshipit-source-id: 2df4092514d58315a6a204411feed99819df5c93
