Summary:
Spliting lock file acquisition from `IdDag::prepare_filesystem_sync` to its own
function.
Useful when looking ahead to split IdDag from IndexedLog.
Reviewed By: quark-zju
Differential Revision: D20316443
fbshipit-source-id: a0fd43439730376920706bb4349ce497f6624335
Summary:
This removes an inline use of the indexedlog indexes.
This is going to be useful when we try to separate IndexedLog specifics from
IdDag functionality.
Reviewed By: quark-zju
Differential Revision: D20316058
fbshipit-source-id: 942a0a71660bb327376c81fd3ac435d002ecca6e
Summary:
Instead of returning `anyhow::Error` wrapping an `ErrorKind` enum
from each Thrift client method, just return an error type specific
to that method. This will make error handling simpler and less
error-prone by removing the need to downcast the returned error.
This diff also removes the `ErrorKind` enums so that we can be sure
that there are no leftover places trying to downcast to them.
(Note: this ignores all push blocking failures!)
Reviewed By: dtolnay
Differential Revision: D20260398
fbshipit-source-id: f0dd96a7b83dd49f6b30948660456539012f82e6
Summary:
The old code does "read, lock, write", which is unsound because after "lock"
the data just read can be outdated and needs a reload.
Reviewed By: xavierd
Differential Revision: D20306137
fbshipit-source-id: a1c29d5078b2d47ee95cf00db8c1fcbe3447cccf
Summary:
I thought the index function could be the bottleneck. However, the Log reading
(xxhash, decoding vlqs) can be much slower for very long entries. Therefore
using bytes as the lag threshold is better. It does leaked the Log
implementation details (how it encodes an entry) to some extend, though.
Reverts D20042045 and D20043116 logically. The lagging calculation is using
the new Index::get_original_meta API, which is easier to verify correctness
(In fact, it seems the old code is wrong - it might skip Index flushes if
sync() is called multiple times without flushing).
This should mitigate an issue where a huge entry (generated by `hg trace`) in
blackbox does not get indexed in time and cause performance regressions.
Reviewed By: DurhamG
Differential Revision: D20286508
fbshipit-source-id: 7cd694b58b95537490047fb1834c16b30d102f18
Summary: This will be used to more reliably detect index lags.
Reviewed By: DurhamG
Differential Revision: D20286518
fbshipit-source-id: c553b6587363a55603b75df12580588e3100e35f
Summary:
This ensures indexes are complete even if index format or definition has been
changed.
Reviewed By: DurhamG
Differential Revision: D20286509
fbshipit-source-id: fcc4ebc616a4501e4b6fd2f1a9826f54f40b99b8
Summary:
This avoids loading all blackbox logs when `init()` gets called multiple times
(for example, once in Rust and once in Python).
Reviewed By: DurhamG
Differential Revision: D20286511
fbshipit-source-id: ef985e454782b787feac90a6249651a882b6552e
Summary: This API has the benefit that it does not trigger loading older logs.
Reviewed By: DurhamG
Differential Revision: D20286512
fbshipit-source-id: 426421691ad1130cdbb2305612d76f18c9f8798c
Summary:
With the new crate-public interfaces and Debug implementations it's possible to
write tests for DagSet. So let's do it.
Reviewed By: sfilipco
Differential Revision: D20242561
fbshipit-source-id: 180e04d9535f79471c79c4307f6ab6e8e8815067
Summary:
Don't restrict constructing a c_api datapack store to only Unix, we can
construct it on Windows too by assuming that their path will be valid UTF-8.
Reviewed By: quark-zju
Differential Revision: D20250718
fbshipit-source-id: 07234b6a71b50c803cfe3b962fa727f57037c919
Summary: This returns the ancestors in the reverser order as the parents method.
Reviewed By: sfilipco
Differential Revision: D20265277
fbshipit-source-id: 83277cee3d8e9070fc56d20d4c1877e6782c22f7
Summary: Those will be reused by nameset::DagSet.
Reviewed By: sfilipco
Differential Revision: D20242563
fbshipit-source-id: 944e9a04aeb15439256ecea64355b67e326e5c89
Summary:
This is useful for `assert_eq!(format!("{:?}", set), "...")` tests.
It will be eventually exposed to Python as `__repr__`, similar to Python's
smartsets.
Reviewed By: sfilipco
Differential Revision: D20242562
fbshipit-source-id: 5373bb180db7cafebf273ace7cf2cb80fbfb8038
Summary:
In the Python world all smartsets have some kind of "debug" information. Let's
do something similar in Rust.
Related code is updated so the test is more readable.
Reviewed By: sfilipco
Differential Revision: D20242564
fbshipit-source-id: 7439c93d82d5d037c7167818f4e1125c5a1e513e
Summary:
Previously, `flush()` will skip writing the file if there are only metadata
changes. Fix it by detecting metadata changes.
This can potentially fix an issue that certain blackbox indexes are empty,
lagging and require scanning the whole log again and again. In that case,
the index itself is not changed (the root radix entry is not changed), but
only the metadata tracking how many bytes in Log the index covered
changed.
Reviewed By: sfilipco
Differential Revision: D20264627
fbshipit-source-id: 7ee48454a92b5786b847d8b1d738cc38183f7a32
Summary:
Using `if cfg!` instead of `#[cfg]` allows for the compiler to understand
that the arguments aren't unused, and silence the warnings.
Reviewed By: quark-zju
Differential Revision: D20242280
fbshipit-source-id: 332dfe17b3a80a1096d15c91c9fb6644bd10e0cd
Summary:
Compiling it on Windows produced a bunch of warning due to
`hgrc_configset_load_path` not being compiled on it. Fixed it so it no longer
depends on Unix specific imports.
Reviewed By: quark-zju
Differential Revision: D20241102
fbshipit-source-id: 3002f961191fbb9bc51aa9ac1154d6d50bd7fe23
Summary:
The `.into_iter()` for this object is being deprecated and won't compile in
the future, fix it now.
Reviewed By: quark-zju
Differential Revision: D20241103
fbshipit-source-id: fdee463ed81cd07a65f3cc4c70a96c88928b3b87
Summary:
While compiling on Windows, this file issues a bunch of warnings, use `if
cfg!` instead of `#[cfg]` to silence them. The behavior is the same, but the
later allows the compiler to recognize that some is not unused.
Reviewed By: quark-zju
Differential Revision: D20241104
fbshipit-source-id: 2cd7f171c7a2f7220cc73bea9be3359260de19b2
Summary:
The change is in theory not necessary. However it improves the reliability on
OS crashes a bit, and can potentially workaround some bugs in filesystems
(as we saw in production where the atomic-written files are empty and the
system didn't crash).
The idea is, the `symlink` syscall does the file creation and "content" writing
together, while there is no way to create a file and write specific content
in one syscall. Note that the C symlink call uses 0-terminated string, and
the Rust stdlib exports it as accepting `Path`. To be safe, we encode binary
or non-utf8 content using `hex`.
For downgrade safety, the write path does not use symlink by default unless
format.use-symlink-atomic-write is set to true. This makes downgrade possible:
the read path is rolled out first, then we can turn on and off the write path.
The indexedlog Rust unit tests and test-doctor.t are migrated to use the new
symlink code paths.
Reviewed By: DurhamG
Differential Revision: D20153864
fbshipit-source-id: c31bd4287a8d29575180fbcf7227d2b04c4c1252
Summary:
This makes it possible to implement atomic_write differently (ex. use a
symlink).
Reviewed By: DurhamG
Differential Revision: D20153865
fbshipit-source-id: 07fa78c2f2dac696668f477c75f65cf70950b73f
Summary:
This makes it clear that `log` is a math concept, not an append-only file like
`Log`.
Reviewed By: DurhamG
Differential Revision: D20149376
fbshipit-source-id: 67d2e9584b15f48759ca9b6dfce4279a5b1365a0
Summary:
Context: https://fb.workplace.com/groups/rust.language/permalink/3338940432821215/
This codemod replaces *all* dependencies on `//common/rust/renamed:futures-preview` with `fbsource//third-party/rust:futures-preview` and their uses in Rust code from `futures_preview::` to `futures::`.
This does not introduce any collisions with `futures::` meaning 0.1 futures because D20168958 previously renamed all of those to `futures_old::` in crates that depend on *both* 0.1 and 0.3 futures.
Codemod performed by:
```
rg \
--files-with-matches \
--type-add buck:TARGETS \
--type buck \
--glob '!/experimental' \
--regexp '(_|\b)rust(_|\b)' \
| sed 's,TARGETS$,:,' \
| xargs \
-x \
buck query "labels(srcs, rdeps(%Ss, //common/rust/renamed:futures-preview, 1))" \
| xargs sed -i 's,\bfutures_preview::,futures::,'
rg \
--files-with-matches \
--type-add buck:TARGETS \
--type buck \
--glob '!/experimental' \
--regexp '(_|\b)rust(_|\b)' \
| xargs sed -i 's,//common/rust/renamed:futures-preview,fbsource//third-party/rust:futures-preview,'
```
Reviewed By: k21
Differential Revision: D20213432
fbshipit-source-id: 07ee643d350c5817cda1f43684d55084f8ac68a6
Summary:
Also patch aho-corasick to fix the issue.
The issue was introduced by [an optimization path](063ca0d253) added in aho-corasick 0.7 series (used by globset 0.4.3).
aho-corasick 0.6.x (globset 0.4.2) are not affected.
The next aho-corasick release (0.7.9) contains the fix.
See https://github.com/BurntSushi/aho-corasick/issues/53 for more context.
Reported by: yns88
Reviewed By: DurhamG
Differential Revision: D20125697
fbshipit-source-id: 592375b43d7ee494bb3e916a1cb11c18f9ebe425
Summary:
Migrate away from some uses of revision numbers.
Some dead code in discovery.py is removed.
I also fixed some test issues when I run tests locally.
Reviewed By: sfilipco
Differential Revision: D20155399
fbshipit-source-id: bfdcb57f06374f9f27be51b0980652ef50a2c8e0
Summary: This makes it possible to use NameIter in py_class.
Reviewed By: sfilipco
Differential Revision: D20020529
fbshipit-source-id: b9147b7dccb38d18d8361b420507fcbe97e01351
Summary: This will be used by commit hash prefix lookup.
Reviewed By: sfilipco
Differential Revision: D20020523
fbshipit-source-id: f2905ddf63098704b08dad8eb48272c3ffba7e25
Summary: Export common types at the top-level of the crate so it's easier to use.
Reviewed By: sfilipco
Differential Revision: D20020526
fbshipit-source-id: e9a0a8bc3cc91f81d0bc74e7530cd4613fc1dd61
Summary: Those just delegate to IdDag for the actual calculation.
Reviewed By: sfilipco
Differential Revision: D20020522
fbshipit-source-id: 272828c520097c993ab50dac6ecc94dc370c8e8b
Summary: This will be used to produce NameSet.
Reviewed By: sfilipco
Differential Revision: D20020519
fbshipit-source-id: abf6d73f2b985b74560d6b5db2800ff25450f02e
Summary: DagSet's SpanSet has fast paths for set operations. Use them.
Reviewed By: sfilipco
Differential Revision: D19912104
fbshipit-source-id: 24b55aa14d03be2f1be59c923e0b8e79d6bcbe6d
Summary: This is similar to hg's fullreposet. It'll be useful as a dummy "subset".
Reviewed By: sfilipco
Differential Revision: D19912108
fbshipit-source-id: 33a95bcb3cf5931a431a1201d1a1f3c627cec7a1
Summary: SortedSet is a wrapper to other sets that marks it as topologically sorted.
Reviewed By: sfilipco
Differential Revision: D19912111
fbshipit-source-id: 2637e8fd29b97f6db0c5bae3f0decd7ac382eeb1
Summary:
Wraps SpanSet + IdMap so it only exposes commit names without ids.
There is no equivalent smartset in Mercurial.
Reviewed By: sfilipco
Differential Revision: D19912112
fbshipit-source-id: 0d257de11527dfa8836065ac94f652730a97a468
Summary: Similar to Mercurial's smartset.baseset. All names are statically known.
Reviewed By: sfilipco
Differential Revision: D19912105
fbshipit-source-id: e4fcf2d59291adb3ca01b3b90f1ac32c65ad7eaa
Summary:
This is an example about how to use the new Bytes type. The performance change
is not obviously visible in benchmarks since the bottleneck is not at the bytes
copying.
Reviewed By: DurhamG
Differential Revision: D19818720
fbshipit-source-id: a431ae206cfa4fa08b2e162a48b3d7cbcd900f7f
Summary: The APIs are compatible so the switch is straightforward.
Reviewed By: DurhamG
Differential Revision: D19818713
fbshipit-source-id: 504e9149567c90eb661804e0dad20580a401aa76
Summary:
D20042045 changes the meaning of "lag_threshold". Update the value in mutation
store accordingly.
Reviewed By: DurhamG
Differential Revision: D20043116
fbshipit-source-id: 154e6dc2aa88ab0a9a9b21929ae5fa6163dcd403
Summary:
Previously indexes are only updated at `sync()` time. This diff makes it so
`open()` can also update lagging indexes. This should make index migration
(ex. D19851355) smoother - indexes are built in time and users suffer less from
the absent of indexes.
Reviewed By: DurhamG
Differential Revision: D20042046
fbshipit-source-id: 20412661a0ca4f5f67b671137c47b6373a42981d
Summary: The logic is currently only used by `sync()`. I'd like to reuse it at `open()`.
Reviewed By: DurhamG
Differential Revision: D20042044
fbshipit-source-id: 5c9734ff68bdcf8f8c8710c6a821b18d3afeaca0
Summary:
This is more friendly for indexedlog users - deciding lag_threshold by number
of entries is easier than by bytes.
Initially, I thought checking `bytes` is cheaper and checking `entries` is more
expensive. However, practically we will have to build indexes for `entires`
anyway. So we do know the number of entries lagging behind.
Reviewed By: DurhamG
Differential Revision: D20042045
fbshipit-source-id: 73042e406bd8b262d5ef9875e45a3fd5f29f78cf
Summary:
This can be useful for users of indexedlog when they want `Bytes` (to get rid
of the lifetime parameter).
This might be useful for storage layer that wants to take the ownership of the
returned bytes.
Reviewed By: xavierd
Differential Revision: D19818714
fbshipit-source-id: cb2d4e7deff921915e07454fee15cb94a3d5c00d
