Summary:
The motivation of this is, LogRotate might copy dirty (non-flushed) entries
from one Log to another, and it cannot preserve the checksum type for those
entries. There are 2 solutions:
- Make `iter_dirty` return checksum type.
- Make checksum type known by Log directly.
The second choice provides a simpler public API. `append_advanced` can be
removed, then `iter_dirty` is still consistent with `iter`. Therefore this
change.
Differential Revision: D14688174
fbshipit-source-id: 09e07d64c886a5ce9bc48dce8e29d036af1c0381
Summary: A later diff adds another field to OpenOptions that Log needs access to.
Differential Revision: D14688171
fbshipit-source-id: 33170a2b74639ba0fd8a9c86207d840fb6427580
Summary: This is the final piece to make space usage bounded.
Differential Revision: D14688179
fbshipit-source-id: a6e0058b9022789fcf036c4427d29eab19144b53
Summary:
If "latest" pointer has changed, we should write to the new "latest" Log,
instead of the stale one.
Differential Revision: D14688180
fbshipit-source-id: eab8df8ddb8f311e472361ecc2b1bc4155f2aba4
Summary:
This API iterates entries that are in-memory only. It is useful to extract
entries and store them elsewhere.
Differential Revision: D14688178
fbshipit-source-id: 6ace51d859ba6886aeb94689f6c45162b9c6958e
Summary: Implement the basic flush logic. Missing bits are listed as TODO items.
Differential Revision: D14688177
fbshipit-source-id: 3613009ec2c216398af6eaff44487a20ceeb97ef
Summary:
The file size will be used to decide whether the Log needs "rotate" in upcoming
changes.
Reviewed By: kulshrax
Differential Revision: D14688169
fbshipit-source-id: b273abcc870b96650d2c76e6e742a3141ce48f13
Summary:
These methods just delegate to `Log` structures. Unfortunately, the key has to
be copied so it can be used by the iterator to query remaining logs.
Differential Revision: D14688172
fbshipit-source-id: fd581f7256031a0622ec0533c84daaab89f9bb82
Summary:
Start implementing the "log rotate" idea by markbt. It is similar to
logrotate, with plain text log files replaced by indexedlog. This
implementation also avoids renaming, which can be troublesome on Windows,
by just increasing the number (ex. to rotate "1/", "2/", create "3/", and
delete "1/", without renaming "2/").
The main use case would be LRU key-value cache on disk.
Reviewed By: kulshrax
Differential Revision: D14688176
fbshipit-source-id: 3bf7917e06386ebf85d8d6deeea850c58f4875e8
Summary:
One of the future need is to open a `Log` without creating it by default. The
newly added `create` option can be disabled to prevent that.
This also changes the code path so we no longer take a directory lock
unconditionally during `open`.
Differential Revision: D14688173
fbshipit-source-id: 88795d5637a1a5135d4014434b2cf828540c0333
Summary:
One of the upcoming changes is to add an option to avoid creating Log on demand
at open time. To avoid `open` being too complicated, add an `OpenOptions` struct.
This is consistent with `index` and `std::fs`.
Differential Revision: D14688175
fbshipit-source-id: bb7f1556a32f1f7b15c64a23c5aee7493dd40ce6
Summary:
This diff fixes the behavior of `skip_subtree` when called on a Leaf. The bug is
that the path is not correctly handled in this case. The name of the file
continues to stay in the path resulting in incorrect path names for all
subsequent calls to `path()`.
The high level perspective is that `skip_subtree` is a no-op in a Leaf node.
To fix, clarify the behavior and improve readability of the code we ad a new
state that handles poping elements from the path.
Durham noticed this bug when reviewing D14347655.
Reviewed By: quark-zju
Differential Revision: D14654557
fbshipit-source-id: 625278366e492a3048dddc44f9234a06d6928b7e
Summary:
It's hard to clone a `Fn`. But `fn` can be cloned. Change the API to use `fn`
instead.
Cloning `IndexDef` allows the same index definition to be used by multiple
Logs. It's used by upcoming diffs.
Differential Revision: D14688181
fbshipit-source-id: 6fda03a5f744dc90ee5d7ad3f36c243602f33510
Summary:
This makes it easier to compare benchmark results between abstractions.
A sample of the result is listed below. Comparing to radixbuf, which is highly
optimized and less flexible, indexedlog is about 10x slower on insertion, and
about 3x slower on lookup.
indexedlog:
index insertion (owned key) 90.201 ms
index insertion (referred key) 81.567 ms
index flush 50.285 ms
index lookup (memory) 25.201 ms
index lookup (disk, no verify) 31.325 ms
index lookup (disk, verified) 46.893 ms
log insertion 18.421 ms
log insertion (no checksum) 12.106 ms
log insertion with index 110.143 ms
log flush 8.783 ms
log iteration (memory) 6.444 ms
log iteration (disk) 6.719 ms
raidxbuf:
index insertion 11.874 ms
index lookup 8.495 ms
Differential Revision: D14635330
fbshipit-source-id: 28b3f33b87f4e882cb3839c37a2a11b8ac80d3e9
Summary:
This is just a trivial test case showing the overhead of xxhash.
log insertion 18.359 ms
log insertion (no checksum) 7.835 ms
Differential Revision: D14635329
fbshipit-source-id: adc2629c0c41aaab48d29d467849e4d96eb01c51
Summary: Change the code to work with newer `rand` crate.
Reviewed By: kulshrax
Differential Revision: D14635328
fbshipit-source-id: 007f6749f2eab781a7dcf7d49b19aff1c81089b4
Summary: Add a function that adds all of the entries from a given iterator into a HistoryPack.
Differential Revision: D14647767
fbshipit-source-id: 29a71b37da86125e14135c40c279bfc8a454b568
Summary:
Computing mutation entries for all local commits is expensive when there are
lots of local draft commits. Ideally we would have an indexed changelog that
would make these lookups fast, but until we have that, put entries in the
mutation store for these commits to take advantage of the fast lookup there.
Reviewed By: quark-zju
Differential Revision: D14566782
fbshipit-source-id: cc3a05715337a510a65d8ff436c59d16d0f0447e
Summary: `std::fs` is only needed for Windows. Do not "use" it on *nix systems.
Reviewed By: sfilipco
Differential Revision: D14634779
fbshipit-source-id: 9fd9a29ae27e13f00b4adbc83a74bd92a1b1658c
Summary:
Change fields in IndexDef to private. Provide a public constructor method and
switch users to use that instead. This makes it possible to change the IndexDef
struct in the future (ex. having extra optional fields about whether the index
is backed by radix tree or something different).
Differential Revision: D14608955
fbshipit-source-id: 62a413268d97ba96b2c4efd2ce67cd4fa0ff4293
Summary: Update Rust toolchain to 1.33.0 with fixes to make our code compatible with 1.33.0.
Reviewed By: Imxset21, kulshrax
Differential Revision: D14608312
fbshipit-source-id: 2d9cf7d01692abaed32f9adffa0e5eb51cfacb4f
Summary:
The configparser tests `hg::tests::test_basic_hgplain` and
`hg::tests::test_hgplainexcept` set different values for `HGPLAIN` and
`HGPLAINEXCEPT`. Since the tests run in parallel and use the same environment,
one of the tests may fail if they run at the same time.
For these tests, create a mutex for the environment and lock it for the
duration of the test, ensuring these tests do not interfere with each other.
Reviewed By: jsgf
Differential Revision: D14615394
fbshipit-source-id: 9f123668d93223655514db2ae34b05354a6b578c
Summary:
The ChaCha RNG has moved from `rand` to `rand_chacha`. Use the new location to
prevent the deprecation warning.
Reviewed By: quark-zju
Differential Revision: D14596397
fbshipit-source-id: e082369d4cf2d4ab42eb37df83a2627b937dcf62
Summary:
Windows disallows rewriting or truncating mmaped files. Fix the tests by
either dropping the mmap, or skipping the test.
Reviewed By: sfilipco
Differential Revision: D14572119
fbshipit-source-id: dccafdc66db3830c2919232d899ba31365120066
Summary:
Windows has 2 kinds of paths - the UNC path (starting with `\\?\`), and paths
most people use (ex. `C:\foo\bar`). The former is more powerful (reserved names
like `nul` can be used), and is the "canonicalized" form as seen by Rust stdlib.
The UNC paths are stricter, though. `/` is not treated as `\` automatically,
`.` and `..` are considered illegal. That is, trying to canonicalize a UNC path
with `..` or `.` will result in an error.
It's possible to get `.`, or `..` into part of an UNC path, by using the
`PathBuf::join` API provided by the Rust stdlib. That is, a legal UNC path stored
in `PathBuf` can become illegal by `join`ing a non-UNC path.
I'm not sure what's the most "clean" fix. Perhaps using two different types to
represent UNC path and non-UNC path in stdlib? But that's definitely not a trivial
(or even practical) change.
For now, just teach the config parser to "friendly try again" by stripping the
UNC prefix and re-canonicalize paths. So it can handle `.` or `..` used by
`%include`.
Reviewed By: sfilipco
Differential Revision: D14568119
fbshipit-source-id: 2a55faa945c8d03574fd56e82d946c9ef7f0138f
Summary:
The `load_or_create_meta` function is subject to filesystem races. Solve it by
always taking a lock.
This hurts performance a little bit. But `open()` should not be in a hot loop.
So it should probably be fine.
Reviewed By: sfilipco
Differential Revision: D14568122
fbshipit-source-id: d9b28555ab94252da4717de709b780b361e1dda7
Summary:
On Windows it's impossible to open (2) a directory. Therefore add a utility
function that creates `lock` file automatically on Windows and open that file
instead.
Reviewed By: sfilipco
Differential Revision: D14568117
fbshipit-source-id: bc7ae7046be654560c38fbd98ec4dd58c071b1dc
Summary:
Previously, `load_or_create_meta` could return without actually creating the
meta file. That leads to problems when `load_or_create_meta` is called a
second time via `flush()`, it rewrites the primary file incorrectly. On Windows,
it will fail to rewrite the primary file.
Fix it by actually writing a meta file before returning.
Reviewed By: sfilipco
Differential Revision: D14568118
fbshipit-source-id: da3ad42bf48a923d732b1719839ca1953bd2b06c
Summary: This diff adds serializable types representing batch requests for the Eden API. Just like the response types, these types must live the Mercurial's `types` crate so that they can be shared between the client and server.
Reviewed By: quark-zju
Differential Revision: D14573332
fbshipit-source-id: c31d718e6a97829ce1acfc25b8853dc3761323a7
Summary: The mock values for the `Node` type are intended to have hashes that consist of a repeated digit (e.g., `1111111111111111111111111111111111111111`). However, since the bytes were specified using a single hex digit instead of two, the hashes were actually like `0101010101010101010101010101010101010101`. This diff fixes the values so they look as expected.
Reviewed By: quark-zju
Differential Revision: D14557546
fbshipit-source-id: 23651d70b9715d2fb77db162f689b87d9d43e5a2
Summary:
https://doc.rust-lang.org/rustdoc/documentation-tests.html#syntax-reference
Rust will treat an indentation of 4 or more spaces as a fenced code block and
attempt to run it as a docblock test
Reviewed By: singhsrb
Differential Revision: D14543987
fbshipit-source-id: 92f78e9e052befba0bd3eea80ac171f651f2fced
Summary:
The main issue is that cargo test fails preventing adding sandcastle
configuration that would run these tests on CI.
Reviewed By: singhsrb
Differential Revision: D14543988
fbshipit-source-id: c299148cce01316fad872b9cf8e15dea6633da48
Summary:
Similarly to the previous change let's make the asyncmutabledatapack force the
Metadata to be present.
Reviewed By: sfilipco
Differential Revision: D14443510
fbshipit-source-id: 26f851e8d38297dcc37410f0df6a69083531d516
Summary:
Now that mutablepacks can only create v1 packfile, we can force the Metadata to
not be optional. The main reason for doing this is to avoid issues where LFS
data is stored without its corresponding LFS flag. This can cause issue down
the line as LFS data will be intepreted as is, instead of being interpreted as
a pointer to the LFS blob.
Reviewed By: sfilipco
Differential Revision: D14443509
fbshipit-source-id: 9e7812017fc1356072278496406648f935024f92
Summary:
The v0 doesn't support flags like whether the data is actually an LFS pointer. Let's simply
forbid creating them.
Reviewed By: quark-zju
Differential Revision: D14443512
fbshipit-source-id: 6ffa2e8fda2b2baba0aae53e749bc9248594a134
Summary:
These last 2 errors are still considered fatal, but shouldn't be and are most
likely transient. Failing to open a packfile that was successfully opened
before can for instance happen when the file is removed by another process, or
if it somehow become corrupted. Failing the removal of the pack-file should no
longer be an issue, but if it fails, we can also ignore it with the reasoning
that the next repack will take care of it.
Reviewed By: sfilipco
Differential Revision: D14441288
fbshipit-source-id: 6c2758c2a88fd5d2d83b55defe3d263ee9f974a1
Summary: In order to send batch responses from the API server for data fetching operations, we need to define the types sent over the wire from within `/scm/hg/lib` so that we can deserialize them from within Mercurial. For ease of use, these types implement `IntoIterator` to allow easily iterating over the content (performing type conversions where needed).
Reviewed By: quark-zju
Differential Revision: D14517259
fbshipit-source-id: 5ee867d8386e6b99cb5b4ed96338aeb7eb6a3e44
Summary: When writing tests, it is often desirable to be able to quickly get a dummy value for a `Node` hash or `Key`. Trying to construct one on the spot can be overly verbose, so let's define some mock values that can be used by tests. This is similar to what Mononoke does (e.g., https://fburl.com/p9u55uye).
Reviewed By: quark-zju
Differential Revision: D14517258
fbshipit-source-id: e3d4cdd60010f44ca681d7a87e6124fe79f8a4c6
Summary: `LooseHistoryEntry` and `PackHistoryEntry` aren't the best names for these types, since the latter is what most users should use, whereas the former should only typically used for data transmission. As such, we should rename these to clarify the intent.
Differential Revision: D14512749
fbshipit-source-id: 5293df89766825077b2ba07224297b958bf46002
Summary:
Corrupted packfiles, or background removal of them could cause repack to fail,
let's simply ignore these transient errors and continue repacking.
Reviewed By: DurhamG
Differential Revision: D14373901
fbshipit-source-id: afe88e89a3bd0d010459975abecb2fef7f8dff6f
Summary:
The spec for both XDG and Mercurial say that when the XDG_CONFIG_HOME variable
is not set, we should default to $HOME/.config.
Windows and macOS also have a designated config folder outside of the home directory.
The `dirs` crate provides consistent access to this folder. I see no harm in looking at config
folders across all operating systems.
Reviewed By: quark-zju
Differential Revision: D14380686
fbshipit-source-id: 5e5a9cd4694aaa49fbc526f4917dc4afdaeb9842
