Summary:
There was an issue where if the prefetch inside cansendtrees failed, it
wouldn't allow it to actually try the operation. This is undesirable, since
prefetch only talks to the server while the actual tree fetch will also attempt
to generate a tree from an old flat manifest.
Ideally we'd have a more unified flow here, where we could have the server let
us know what nodes it couldn't find, then the client could try other options for
the remaining nodes, but that requires significantly more refactoring.
Reviewed By: quark-zju
Differential Revision: D7450662
fbshipit-source-id: a023f27ee4b74786633e4dce7e62f3d9604c2b7f
Summary:
It further slows down lookups, even when checksum is disabled, since even a
`is_none()` check is not free:
index insertion 4.697 ms
index flush 3.764 ms
index lookup (memory) 2.878 ms
index lookup (disk, no verify) 3.564 ms
index lookup (disk, verified) 7.788 ms
The "verified" version basically needs 2x time due to more memory lookups.
Unfortunately this means eventual lookup performance will be slower than
gdbm, but insertion is still much faster. And the index still has a better
locking properties (lock-free read) that gdbm does not have.
With correct time complexity (no O(len(changelog)) index-only operations for
example), I'd expect it's rare for the overall performance to be bounded by
index performance. Data integrity is more important.
With a larger number of nodes, ex. 2M 20-byte strings: inserting to memory
takes 1.4 seconds, flushing to disk takes 0.9 seconds, looking up without
checksum takes 0.9 seconds, looking up with checksum takes 1.7 seconds.
Reviewed By: DurhamG
Differential Revision: D7440248
fbshipit-source-id: 020e5204606f9f0a4f68843a491009a6a6f75751
Summary:
This is in the critical path for lookup, and has very visible performance
penalty:
index insertion 3.923 ms
index flush 3.921 ms
index lookup (memory) 1.070 ms
index lookup (disk, no verify) 1.980 ms
index lookup (disk, verified) 5.206 ms
Reviewed By: DurhamG
Differential Revision: D7440252
fbshipit-source-id: 49540f974faff1cdd0603a72328f141ccd054ee2
Summary:
Previously checksum is only for `MemRoot`, now it's for all `Mem` structs.
Since `Mem*` structs are not frequently used in the normal lookup code path,
there is no visible performance change.
Reviewed By: DurhamG
Differential Revision: D7440253
fbshipit-source-id: 945f5a8c38d228f59190a487b0cf6dbc5daac4f7
Summary:
The type will be used all over the place and may make `rustfmt` wrap lines.
Use a shorter type to make it slightly cleaner.
Reviewed By: DurhamG
Differential Revision: D7436338
fbshipit-source-id: ecaada23916a22658f65669b748632a077e60df2
Summary:
This only affects `Index::open` right now. So it's a one time check and does
not affect performance.
Reviewed By: DurhamG
Differential Revision: D7436341
fbshipit-source-id: 30313064bf2ea50320ac744fc18c03bff4b12c89
Summary:
Add `ChecksumTable` to the `Index` struct. But it's not functional yet.
The checksum will mainly affect "index lookup (disk)" case. Add another
benchmark for showing the difference with checksum on and off. They do not
have much difference right now:
index insertion 3.756 ms
index flush 3.469 ms
index lookup (memory) 0.990 ms
index lookup (disk, no verify) 1.768 ms
index lookup (disk, verified) 1.766 ms
Reviewed By: DurhamG
Differential Revision: D7436339
fbshipit-source-id: 60a6554a2c96067a53ce9e1753cd51d0d61c0bea
Summary:
The minibench framework does not provide benchmark filtering. So let's
separate benchmarks using different entry points.
Reviewed By: DurhamG
Differential Revision: D7440250
fbshipit-source-id: 11e7790a5074ebf4c08e33c312a490a66a921926
Summary:
The "clone" benchmarks were added to be subtracted from "lookup" to
workaround the test framework limitation.
The new minibench framework makes it easier to exclude preparation cost.
Therefore the clone benchmarks are no longer needed.
index insertion 3.881 ms
index flush 3.286 ms
index lookup (memory) 0.928 ms
index lookup (disk) 1.685 ms
"index lookup (memory)" is basically "index lookup (memory)" minus
"index clone (memory)" in previous benchmarks.
Reviewed By: DurhamG
Differential Revision: D7440251
fbshipit-source-id: 0e6a1fb7ee64f9a393ee9ada4db6e6eb052e20bf
Summary:
See the previous minibench diff for the motivation.
"failure" was removed from build dependencies since it's not used yet.
Run benchmark a few times. It seems the first several items are less stable
due to possibly warming up issues. Otherwise the result looks good enough.
The test also compiles and runs much faster.
```
base16 iterating 1M bytes 0.921 ms
index insertion 4.804 ms
index flush 5.104 ms
index lookup (memory) 2.929 ms
index lookup (disk) 1.767 ms
index clone (memory) 2.036 ms
index clone (disk) 0.010 ms
base16 iterating 1M bytes 0.853 ms
index insertion 4.512 ms
index flush 4.717 ms
index lookup (memory) 2.907 ms
index lookup (disk) 1.755 ms
index clone (memory) 1.856 ms
index clone (disk) 0.010 ms
base16 iterating 1M bytes 1.525 ms
index insertion 4.577 ms
index flush 4.901 ms
index lookup (memory) 2.800 ms
index lookup (disk) 1.790 ms
index clone (memory) 1.794 ms
index clone (disk) 0.010 ms
base16 iterating 1M bytes 0.768 ms
index insertion 4.486 ms
index flush 4.918 ms
index lookup (memory) 2.658 ms
index lookup (disk) 1.721 ms
index clone (memory) 1.763 ms
index clone (disk) 0.010 ms
base16 iterating 1M bytes 0.732 ms
index insertion 4.489 ms
index flush 4.792 ms
index lookup (memory) 2.689 ms
index lookup (disk) 1.739 ms
index clone (memory) 1.850 ms
index clone (disk) 0.009 ms
base16 iterating 1M bytes 1.124 ms
index insertion 7.188 ms
index flush 4.888 ms
index lookup (memory) 2.829 ms
index lookup (disk) 1.609 ms
index clone (memory) 2.642 ms
index clone (disk) 0.010 ms
base16 iterating 1M bytes 1.055 ms
index insertion 4.683 ms
index flush 4.996 ms
index lookup (memory) 2.782 ms
index lookup (disk) 1.710 ms
index clone (memory) 1.802 ms
index clone (disk) 0.009 ms
```
Reviewed By: DurhamG
Differential Revision: D7440249
fbshipit-source-id: 0f946ab184455acd40c5a38cf46ff94d9e3755c8
Summary:
The dirty -> non-dirty offset mapping can be optimized using a dedicated
"map" type that is backed by `vec`s, because dirty offsets are continuous
per type.
This makes "flush" significantly faster:
```
index flush time: [5.8808 ms 6.1800 ms 6.4813 ms]
change: [-62.250% -59.481% -56.325%] (p = 0.00 < 0.05)
Performance has improved.
```
Reviewed By: DurhamG
Differential Revision: D7422832
fbshipit-source-id: 9ab8a70d1663155941dae5b4f02f7452f5e3cadf
Summary:
It seems to improve the performance a bit:
```
index insertion time: [5.4643 ms 5.6818 ms 5.9188 ms]
change: [-24.526% -17.384% -10.315%] (p = 0.00 < 0.05)
Performance has improved.
```
Reviewed By: DurhamG
Differential Revision: D7422831
fbshipit-source-id: fc1c72f402258db7e189cd8724583757d48affb7
Summary:
For key entries, the key is immutable once stored. So just use `Box<[u8]>`.
It saves a `usize` per entry. On 64-bit platform, that's a lot.
Performance is slightly improved and it catches up with D7404532 before
typed offset refactoring now:
index insertion time: [6.1852 ms 6.6598 ms 7.2433 ms]
index flush time: [15.814 ms 16.538 ms 17.235 ms]
index lookup (memory) time: [3.7636 ms 3.9403 ms 4.1424 ms]
index lookup (disk) time: [1.9413 ms 2.0366 ms 2.1325 ms]
index clone (memory) time: [2.6952 ms 2.9221 ms 3.0968 ms]
index clone (disk) time: [5.0296 us 5.2862 us 5.5629 us]
Reviewed By: DurhamG
Differential Revision: D7422837
fbshipit-source-id: 4aabfdc028aefb8e796803e103f0b2e4965f84e6
Summary:
Previously, both `value` and `link` are optional in `insert_advanced`.
This diff makes `value` required.
`maybe_create_link_entry` becomes unused and removed.
No visible performance change.
Reviewed By: DurhamG
Differential Revision: D7422838
fbshipit-source-id: 8d7d3cc1cc325f6fea7e8ce996d0a43d3ee49839
Summary:
Also add an IMM test to tease out working-copy vs. non-working-copy issues.
Also add some newlines to code stolen from fbcode.
Reviewed By: DurhamG
Differential Revision: D7432333
fbshipit-source-id: 029ccd8aeec7f0e2c380da41e7d78b433a275af3
Summary:
This is a large refactoring that replaces `u64` offsets with strong typed
ones.
Tests about serialization are removed since they generate illegal data that
cannot pass type check.
It seems to slow down the code a bit, comparing with D7404532. But there are
still room to improve.
index insertion time: [6.9395 ms 7.3863 ms 7.7620 ms]
index flush time: [15.949 ms 17.965 ms 20.246 ms]
index lookup (memory) time: [3.6212 ms 3.8855 ms 4.1923 ms]
index lookup (disk) time: [2.2496 ms 2.4649 ms 2.8090 ms]
index clone (memory) time: [2.7292 ms 2.9399 ms 3.2055 ms]
index clone (disk) time: [4.9239 us 5.5928 us 6.3167 us]
Reviewed By: DurhamG
Differential Revision: D7422833
fbshipit-source-id: 7357cb0f4f573f620e829c5e300cd423619dbd62
Summary:
I got errors running `histedit --abort` in the code path:
quark [1] % hg histedit --abort
2 files updated, 0 files merged, 0 files removed, 0 files unresolved
saved backup bundle to .hg/strip-backup/098a5bf950b2-78da64d6-backup.hg
saved backup bundle to .hg/strip-backup/accd7866dec2-599f621b-backup.hg
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/mercurial/scmutil.py", line 159, in callcatch
return func()
File "/usr/lib/python2.7/site-packages/mercurial/dispatch.py", line 340, in _runcatchfunc
return _dispatch(req)
File "/usr/lib/python2.7/site-packages/mercurial/dispatch.py", line 944, in _dispatch
cmdpats, cmdoptions)
File "/usr/lib/python2.7/site-packages/hgext/remotefilelog/__init__.py", line 458, in runcommand
return orig(lui, repo, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/hgext/journal.py", line 85, in runcommand
return orig(lui, repo, cmd, fullargs, *args)
File "/usr/lib/python2.7/site-packages/hgext/undo.py", line 118, in _runcommandwrapper
result = orig(lui, repo, cmd, fullargs, *args)
File "/usr/lib/python2.7/site-packages/hgext/fastmanifest/__init__.py", line 202, in _logonexit
r = orig(ui, repo, cmd, fullargs, *args)
File "/usr/lib/python2.7/site-packages/hgext/perftweaks.py", line 326, in _tracksparseprofiles
res = runcommand(lui, repo, *args)
File "/usr/lib/python2.7/site-packages/hgext/perftweaks.py", line 313, in _trackdirstatesizes
res = runcommand(lui, repo, *args)
File "/usr/lib/python2.7/site-packages/hgext/fbamend/hiddenoverride.py", line 92, in runcommand
result = orig(lui, repo, cmd, fullargs, *args)
File "/usr/lib/python2.7/site-packages/mercurial/dispatch.py", line 693, in runcommand
ret = _runcommand(ui, options, cmd, d)
File "/usr/lib/python2.7/site-packages/mercurial/dispatch.py", line 952, in _runcommand
return cmdfunc()
File "/usr/lib/python2.7/site-packages/mercurial/dispatch.py", line 941, in <lambda>
d = lambda: util.checksignature(func)(ui, *args, **strcmdopt)
File "/usr/lib/python2.7/site-packages/hgext/remotenames.py", line 698, in exhistedit
ret = orig(ui, repo, *args, **opts)
File "/usr/lib/python2.7/site-packages/hgext/histedit.py", line 1033, in histedit
release(state.lock, state.wlock)
File "/usr/lib/python2.7/site-packages/mercurial/lock.py", line 329, in release
lock.release()
File "/usr/lib/python2.7/site-packages/mercurial/lock.py", line 311, in release
self.releasefn()
File "/usr/lib/python2.7/site-packages/hgext/fsmonitor/__init__.py", line 878, in staterelease
l.stateupdate.exit()
File "/usr/lib/python2.7/site-packages/hgext/fsmonitor/__init__.py", line 751, in exit
self.repo, self.oldnode, self.newnode)
File "/usr/lib/python2.7/site-packages/hgext/fsmonitor/__init__.py", line 787, in calcdistance
anc = repo.changelog.ancestor(oldnode, newnode)
File "/usr/lib/python2.7/site-packages/mercurial/revlog.py", line 1139, in ancestor
a, b = self.rev(a), self.rev(b)
File "/usr/lib/python2.7/site-packages/mercurial/changelog.py", line 360, in rev
r = super(changelog, self).rev(node)
File "/usr/lib/python2.7/site-packages/mercurial/revlog.py", line 559, in rev
raise LookupError(node, self.indexfile, _('no node'))
LookupError: 00changelog.i@67b8abf62104: no node
abort: 00changelog.i@67b8abf62104: no node!
In case real strip happens, and we are in some nested "update state" situation, the
nodes might be gone:
enter lock state with src: node1, dest: node2
real strip node1 or node2 <- this is possible
exit lock state - node2 is missing
Reviewed By: wez
Differential Revision: D7462306
fbshipit-source-id: 133252583519bc6916a00df4a4a82b36591fb8a5
Summary:
When the fsmonitor state gets invalidated and a full repo walk happens (e.g. after `rebase --abort`), it's a very frustrating experience because Mercurial appears to hang without any output until the walk is done (1-2min for my www clone).
As we did for "querying watchman", let's add a spinner here so people know what's going on, other than "Mercurial is slow". It also will make it easier to claim impact to whoever helps fix our fsmonitor invalidation story[1], since they can tell users "I fixed the 'full repo walk' issue".
Reviewed By: DurhamG, quark-zju
Differential Revision: D7448662
fbshipit-source-id: 011eb742bf388328b3cceb2762681b4f2e9a4eb1
Summary:
Previously we would throw an unhandled exception which would cause
stderr output on the client. Since we sometimes want to try the server silently
from the client, let's return errors as a bundle2 part that we can handle more
gracefully.
Reviewed By: quark-zju
Differential Revision: D7448101
fbshipit-source-id: f0c5d0af56e718f0403ed9a18d66ad8be150d5b8
Summary:
Data stores have already migrated to MissingNodesError instead of
KeyError, so let's move metadatastore as well. This provides better error
messages and more specific catching.
Reviewed By: phillco
Differential Revision: D7448103
fbshipit-source-id: 33d0f267545abd7d4063d2b344a93d26aff76d81
Summary:
Since error parts pass the message and hint as parameters instead of
payload, they are limited to 255 characters. Let's add a helper function to
enforce this.
Reviewed By: quark-zju
Differential Revision: D7448104
fbshipit-source-id: 33d47a21e7159b6c4bd72cad9669568b92a51e34
Summary:
In an in-memory merge, if a commit only changed the flags of a file, and that file also never got written to during the merge, the IMM could fail and cause it to restart.
The reason is pretty simple: `setflags()` sets `cache[flags]` but not `cache[data]`, as it doesn't have any new data to store. In that case, calls to read the data should to fall-through to the underlying `p1` context.
Indeed, proper logic to do that already exists in `overlayworkingctx.data(path)` and `flags(path)`. The problem is that `tomemctx()` was reading from the cache directly, which is problematic and unhygenic. So let's just change it to call the proper functions, which also fixes the bug.
Reviewed By: DurhamG
Differential Revision: D7447640
fbshipit-source-id: 1625ef82ad2683c6a72059a0944fd5e336d3ec3a
Summary:
`mercurial.rust` needs to be an explicit package for importing from it to
work.
(Note: this ignores all push blocking failures!)
Reviewed By: DurhamG
Differential Revision: D7449336
fbshipit-source-id: 36113a6a2e4fbb12d99b538b1c06cfa1023a035b
Summary:
This one actually broke IMM auto-restarting.
For reasons unknown to humanity, the rebase collapse message is saved (and read) to the same place (.hg/last-message.txt) that the commit message is often written to (before each commit in repo.commitctx(), for example). Thus, `rbsrt.collapsemsg` can sometimes be read from this file incorrectly.
If a rebase fails to commit and you abort, and retry, the abort will read `.hg/last-message.txt` and set the `rebasestate`'s collapsemsg to it. Then, the retry rebase will fail because it has a collapsemsg on its state, but no --collapse.
Anyway, two obvious fixes: let's just only read this message if collapse mode has been set, and let's use a fresh runtime for the non-IMM rebase. I don't think it's safe to use a statefile after abort() has been called on it.
Reviewed By: singhsrb
Differential Revision: D7441659
fbshipit-source-id: f5f8920c3f1d26c8ced083b1c0e375e09fe3248b
Summary:
If we get an exception before we write the rebasestate, aborting will fail since there's nothing to abort, preventing us from actually trying to re-run the rebase.
Let's just catch any errors in the abort stage, since the restart is the most important thing there.
Reviewed By: singhsrb
Differential Revision: D7441313
fbshipit-source-id: b019d2c3c36d3bb191447c0315739fd2e86c93fb
Summary:
Use the new gitignore matcher powered by Rust.
The hgignore matcher has some laziness, but is not tree-aware - with N
"hgignore" files loaded, it needs O(N) time to match. The gitignore matcher
is tree-aware and backed by native code with decent time complexity.
We have been maintaining a translation script that collects all gitignores,
generate hgignore files with very long regexp for them. That script has
issues with sparse recently. This diff allows us to remove those generated
hgignore files from the repo.
Note: fsmonitor state does not contain ignored files. And ignore
invalidation is generally broken in fsmonitor (it only checks top-level
.hgignore). That means, once a file is ignored, it cannot be "unignored" by
just removing the matched pattern from ".gitignore". The file has to be
"touched" or so.
Reviewed By: markbt
Differential Revision: D7319608
fbshipit-source-id: 1763544aedb44676413efb6d14ffd3917ed3b1cd
Summary: Build the new Rust matcher with both buck and setup.py
Reviewed By: markbt
Differential Revision: D7319607
fbshipit-source-id: c5944a28602495a9127acb20b59eb95632a9a1f5
Summary:
It only contains a `gitignorematcher` which exposes `GitignoreMatcher`
features to Python.
Reviewed By: markbt
Differential Revision: D7319605
fbshipit-source-id: 846964a551813f9b0933bc30f4a0ba3f85362944
Summary:
The "pathmatcher" crate is intended to eventually cover more "matcher"
abilities so all Python "matcher" related logic can be handled by Rust.
For now, it only contains a gitignore matcher.
The gitignore matcher is designed to work in a repo (no need to create
multiple gitignore matchers for a repo from a higher layer), and be lazy
i.e. be tree-aware, and do not parse ".gitignore" unless necessary.
Worth mentioning that the gitignore logic provided by the "ignore" crate
seems decent in time complexity - it uses regular expression, which uses state
machines to achieve "testing against multiple patterns at once", instead of
testing patterns one-by-one like what git currently does.
Note: The "ignore" crate provides a nice "Walker" interface but that does
not fit very well with the required laziness here. So the walker interface
is not used.
Reviewed By: markbt
Differential Revision: D7319609
fbshipit-source-id: ebd131adf45a38f83acdf653f5e49d0624012152
Summary:
- Support 8 color mode, since some terminals (emacs) do not support 16
colors (i.e. "bright*" colors have no effects).
- Detect emacs and use 8 color mode automatically.
- Read terminfo colors. Respect it if it says the terminal supports more
colors. Suggested by wez.
- Read HGCOLORS. Respect it unconditionally. Change run-tests.py to set it.
- Change diff related colors to also fallback to 8 colors since "bright"
colors are not guaranteed available.
Differential Revision: D7432965
fbshipit-source-id: da75829f856b4de737b946af72d24ff5351026cb
Summary:
Adds two new template functions:
* `simpledate` formats recent dates in a human-friendly way, for example `Today
at 12:00`, `Wednesday at 07:30`, or `Jan 05 at 23:45`. The timezone used
can be overridden.
* `smartdate` shows different results depending on whether a date is recent
or not, where "recent" is defined by a threshold parameter.
Reviewed By: quark-zju
Differential Revision: D7123734
fbshipit-source-id: 7b68207e4debc85c0bfa72bba9b375a4aa11d7c6
Summary: This makes it clear the code has different code paths for on-disk entries.
Reviewed By: DurhamG
Differential Revision: D7422836
fbshipit-source-id: 018fa0e2c20682d4e1beba99f3307550e1f40388
Summary:
Add benchmarks inserting / looking up 20K entries.
Benchmark results on my laptop are:
index insertion time: [6.5339 ms 6.8174 ms 7.1805 ms]
index flush time: [15.651 ms 16.103 ms 16.537 ms]
index lookup (memory) time: [3.6995 ms 4.0252 ms 4.3046 ms]
index lookup (disk) time: [1.9986 ms 2.1224 ms 2.2464 ms]
index clone (memory) time: [2.5943 ms 2.6866 ms 2.7749 ms]
index clone (disk) time: [5.2302 us 5.5477 us 5.9518 us]
Comparing with highly optimized radixbuf:
index insertion time: [991.89 us 1.1708 ms 1.3844 ms]
index lookup time: [863.83 us 945.69 us 1.0304 ms]
Insertion takes 6x time. Lookup from memory takes 1.4x time, from disk takes
2.2x time. Flushing is the slowest - it needs 16x radixbuf insertion time.
Note: need to subtract "clone" time from "lookup" to get meaningful values
about "lookup". This cannot be done automatically due to the limitation of the
benchmark framework.
Although it's slower than radixbuf, the index is still faster than gdbm and
rocksdb. Note: the index does less than gdbm/rocksdb since it does not return
a `[u8]`-ish which requires extra lookups. So it's not a very fair comparison.
gdbm insertion time: [69.607 ms 75.102 ms 79.334 ms]
gdbm lookup time: [9.0855 ms 9.8480 ms 10.637 ms]
gdbm prepare time: [110.35 us 120.40 us 135.63 us]
rocksdb insertion time: [117.96 ms 123.42 ms 127.85 ms]
rocksdb lookup time: [24.413 ms 26.147 ms 28.153 ms]
rocksdb prepare time: [3.8316 ms 4.1776 ms 4.5039 ms]
Note: Subtract "prepare" from "insertion" to get meaningful values.
Code to benchmark rocksdb and gdbm:
```
extern crate criterion;
extern crate gnudbm;
extern crate rand;
extern crate rocksdb;
extern crate tempdir;
use criterion::Criterion;
use gnudbm::GdbmOpener;
use rand::{ChaChaRng, Rng};
use rocksdb::DB;
use tempdir::TempDir;
const N: usize = 20480;
/// Generate random buffer
fn gen_buf(size: usize) -> Vec<u8> {
let mut buf = vec![0u8; size];
ChaChaRng::new_unseeded().fill_bytes(buf.as_mut());
buf
}
fn criterion_benchmark(c: &mut Criterion) {
c.bench_function("rocksdb prepare", |b| {
b.iter(move || {
let dir = TempDir::new("index").expect("TempDir::new");
let _db = DB::open_default(dir.path().join("a")).unwrap();
});
});
c.bench_function("rocksdb insertion", |b| {
let buf = gen_buf(N * 20);
b.iter(move || {
let dir = TempDir::new("index").expect("TempDir::new");
let db = DB::open_default(dir.path().join("a")).unwrap();
for i in 0..N {
db.put(&&buf[20 * i..20 * (i + 1)], b"v").unwrap();
}
});
});
c.bench_function("rocksdb lookup", |b| {
let dir = TempDir::new("index").expect("TempDir::new");
let db = DB::open_default(dir.path().join("a")).unwrap();
let buf = gen_buf(N * 20);
for i in 0..N {
db.put(&&buf[20 * i..20 * (i + 1)], b"v").unwrap();
}
b.iter(move || {
for i in 0..N {
db.get(&&buf[20 * i..20 * (i + 1)]).unwrap();
}
});
});
c.bench_function("gdbm prepare", |b| {
let buf = gen_buf(N * 20);
b.iter(move || {
let dir = TempDir::new("index").expect("TempDir::new");
let _db = GdbmOpener::new().create(true).readwrite(dir.path().join("a")).unwrap();
});
});
c.bench_function("gdbm insertion", |b| {
let buf = gen_buf(N * 20);
b.iter(move || {
let dir = TempDir::new("index").expect("TempDir::new");
let mut db = GdbmOpener::new().create(true).readwrite(dir.path().join("a")).unwrap();
for i in 0..N {
db.store(&&buf[20 * i..20 * (i + 1)], b"v").unwrap();
}
});
});
c.bench_function("gdbm lookup", |b| {
let dir = TempDir::new("index").expect("TempDir::new");
let mut db = GdbmOpener::new().create(true).readwrite(dir.path().join("a")).unwrap();
let buf = gen_buf(N * 20);
for i in 0..N {
db.store(&&buf[20 * i..20 * (i + 1)], b"v").unwrap();
}
b.iter(move || {
for i in 0..N {
db.fetch(&&buf[20 * i..20 * (i + 1)]).unwrap();
}
});
});
}
criterion_group!{
name=benches;
config=Criterion::default().sample_size(20);
targets=criterion_benchmark
}
criterion_main!(benches);
```
Reviewed By: DurhamG
Differential Revision: D7404532
fbshipit-source-id: ff39f520b78ad1b71eb36970506b313bb2ff426b
Summary:
This will be useful for benchmarks - prepare an index as a template, and
clone it in the tests.
Reviewed By: DurhamG
Differential Revision: D7422835
fbshipit-source-id: 190bbdee7cb7c1526274b4d4dab07af4984b5df6
Summary:
The latest rustfmt disagrees about the order of `std::io` imports. Move the
troublesome line to a separate group so both the old and new rustfmt agress
on the format.
Reviewed By: DurhamG
Differential Revision: D7422834
fbshipit-source-id: 9f5289ef2af1a691559fe691e121190f6d845162
Summary:
The old `rustc-test` crate no longer works. There is an upstream
bug report at https://github.com/servo/rustc-test/issues/7.
This change makes it possible to compare radixbuf performance
with the new index.
Reviewed By: DurhamG
Differential Revision: D7404531
fbshipit-source-id: 515e732a65388db4c865c7b139d0f57ead76f788
Summary:
Radix entries need to be written in an reversed order given the order they
are added to the vector.
Reviewed By: DurhamG
Differential Revision: D7404530
fbshipit-source-id: 403189b5c0fa6f21183e62eea04ce4ce7c4e1129
Summary: Those little read and write helpers are used in the next diff.
Reviewed By: DurhamG
Differential Revision: D7377214
fbshipit-source-id: c6e2d240334c11a0b08b15cd7d5c114b6f4d8ace
Summary:
Add a helper function `peek_key_entry_content` that checks key type and
return the key content.
Reviewed By: DurhamG
Differential Revision: D7377211
fbshipit-source-id: 0ce509aba30309373a709cf5fbcb909dd80471dc
Summary:
Implement insertion when there is no need to split a leaf entry.
The API may be subject to change if we want other value types. For now, it's
better to get something working and can be benchmarked so we have data about
performance impact with new format changes.
Reviewed By: DurhamG
Differential Revision: D7343423
fbshipit-source-id: 9761f72168046dbafcb00883634aa7ad513a522b
Summary:
Like the `peek_` family of helper methods. Those methods handles writing
data for both dirty (in-memory) and non-dirty (on-disk) cases. They will
be used in the next diff.
Reviewed By: DurhamG
Differential Revision: D7377208
fbshipit-source-id: f458a20da4bb7808f37daeed3077be2f7e90a9df
Summary:
Add code to print out Index's on-disk and in-memory entries in
human-friendly form. This is useful for explaining its internal state, so it
could be used in tests.
Reviewed By: DurhamG
Differential Revision: D7343427
fbshipit-source-id: 706a35404ea42c413657b389166729f8dd1315a3
Summary:
Offset stored in it needs to be translated, as done in other types of
entries. I forgot it.
Reviewed By: DurhamG
Differential Revision: D7404528
fbshipit-source-id: fb09a9c3052ddfe8f8016440290062084d5d8b03
Summary:
This is a low-level API that follows the base16 sequence of a key, and
return potentially matched `LinkOffset`.
Reviewed By: DurhamG
Differential Revision: D7343424
fbshipit-source-id: 38f260064d1a23695a28dda6f7dc921f88c7fccc
Summary:
Add a bunch of helper methods to "peek" data inside all kinds of entries.
They will be used in the next diff.
The benefit of those helper methods is they handle both dirty offsets and
non-dirty offsets transparently. Previously I have tried to always parse
on-disk entries into in-memory ones and stored them in a hashmap cache.
But that turned to have too much overhead so always reading from disk is
more desirable. It seems to provide at least 2x perf improvement from my
previous quick test.
Reviewed By: DurhamG
Differential Revision: D7377207
fbshipit-source-id: 1b393f1fe64c1d54b986ba7c3b03c790adb694d4
Summary:
The `non_dirty` helper method enforces the offset to be a non-dirty one.
It will be used frequently for checking offsets read from the disk, since
the on-disk offsets shouldn't have any reference to dirty (in-memory)
entries.
Reviewed By: DurhamG
Differential Revision: D7377209
fbshipit-source-id: c6c381c065d3ba8aaa65698224e4778b86edbc4a
