Summary: The API will be used in Python bindings to avoid running Python in background threads.
Reviewed By: sfilipco
Differential Revision: D25513055
fbshipit-source-id: a108b55115271a256c0d43e0ff7b82c0b209be81
Summary:
Previously only `iterctx` does prefetch. Make `__iter__` do prefetch via `iterctx`.
The old `__iter__` without prefetching was renamed to `iterrev`.
Reviewed By: sfilipco
Differential Revision: D24365404
fbshipit-source-id: db5c687066794257719bb64c673dc384b5460ff1
Summary:
Now smartset has a reference to repo. It does not need `repo` from external
source.
Reviewed By: sfilipco
Differential Revision: D24365405
fbshipit-source-id: 8a43697b7b84a8a41691ed8f095c271107a90f16
Summary:
This will make `__iter__` to be able to do proper prefetch, or make it possible
for `__iter__` to return node instead of rev. To avoid cycles, weakref is used.
The smartset types are used widely. It's hard to migrate all callsites at once.
For now, `repo` is optional. Later, it will be required.
Reviewed By: DurhamG
Differential Revision: D24365400
fbshipit-source-id: 5dd40e3d930893c39f16da8f3169b026c8933bd2
Summary:
The test is one of the most fragile tests (ex. it has traceback that is
fragile). Since we don't support 3rd party extensions the test is less
meaningful. Let's remove it.
Reviewed By: singhsrb
Differential Revision: D25514727
fbshipit-source-id: cc3491597ba38ad84c4c2b69a275da6505ae2207
Summary:
It has a high chance of failing with:
```
$ hg --config extensions.b=b.py --config 'pager.pager=head -1' init foo2
+ close failed in file object destructor:
+ IOError: [Errno 9] Bad file descriptor
line1
```
when running with chg together with other tests.
Reviewed By: singhsrb
Differential Revision: D25514728
fbshipit-source-id: d9d2adea97bc6a436341be6f6d9aac43cb78c90a
Summary:
This is a bit tricky - look-ahead does not work because the content of commit
texts decide whether the commit is in the resulting set or not.
Add a special case for `ancestorsaged` so it does not end up with 1-by-1
fetching.
Reviewed By: sfilipco
Differential Revision: D24324797
fbshipit-source-id: 844ac6a6637e25900eea00d99704a59dfc7a0345
Summary:
Some revset function need commit text data to be efficient. Prefetch commit
text explicitly for them.
Reviewed By: DurhamG
Differential Revision: D24322227
fbshipit-source-id: 49c0b4f8cd9d9a744e10d169a1b1247faa8edd8e
Summary: Made unnecessary by using 2018 edition for rust_bindgen_library targets (D25441322).
Reviewed By: jsgf
Differential Revision: D25441329
fbshipit-source-id: d00aaad09451c77c6d05ed5d671468a481ce4e25
Summary:
Some of our tests are marked as flaky because of timeouts. They usually run
under timeout but because it's integration tests the times vary a lot.
The `#requre slow` is marking those tests as slow giving them more time to tun.
Reviewed By: krallin
Differential Revision: D25495555
fbshipit-source-id: 02bc3755992f56f5e743835318cf1233ab7c623d
Summary:
Now that the mapping is separated from BonsaiDerivable, it becomes clear where
batch derivation is incorrectly using the default mapping, rather than the
mapping that has been provided for batch-derivation.
This could mean, for example, if we are backfilling a v2 of these derived
data types, we could accidentally base them on a v1 parent that we obtained
from the default mapping.
Instead, ensure that we don't use `BonsaiDerived` and thus can't use the
default mapping.
Reviewed By: krallin
Differential Revision: D25371963
fbshipit-source-id: fb71e1f1c4bd7a112d3099e0e5c5c7111d457cd2
Summary:
The backfiller may read or write to the blobstore too quickly. Apply QPS
limits to the backfill batch context to keep the read or write rate acceptable.
Reviewed By: ahornby
Differential Revision: D25371966
fbshipit-source-id: 276bf2dd428f7f66f7472aabd9e943eec5733afe
Summary:
The common case of limiting blobstore rates using a leaky bucket rate limiter
is cumbersone to set up. Create a convenience method to do it more easily.
Reviewed By: ahornby
Differential Revision: D25438685
fbshipit-source-id: 821eda7bd0ddf71f22378c1b23e66b6d3f6454e7
Summary:
When fetching many derived data mappings, the use of `FuturesUnordered` means
we may fetch many blobs concurrently, which may overload the blobstore.
Switch to using `buffered` to reduce the number of concurrent blob fetches.
Reviewed By: ahornby
Differential Revision: D25371965
fbshipit-source-id: 30417e86bc33defbb821f214a5520ab1b8a8c18c
Summary:
Large batches with parallel derivation can cause problems in large repos.
Allow control of the batch size so that it can be reduced if needed.
Reviewed By: krallin
Differential Revision: D25401205
fbshipit-source-id: 88a76a7745c34e4e34bc9b3ea9228bd5dad857f6
Summary:
Re-introduce parallel backfilling of changesets in a batch using `batch_derive`,
however keep it under the control of a flag, so we can enable or disable it as
necessary.
Reviewed By: ahornby
Differential Revision: D25401207
fbshipit-source-id: f9aeef3415be48fc03220c18fa547e05538ed479
Summary:
Change derived data config to have "enabled" config and "backfilling" config.
The `Mapping` object has the responsibility of encapsulating the configuration options
for the derived data type. Since it is only possible to obtain a `Mapping` from
appropriate configuration, ownership of a `Mapping` means derivation is permitted,
and so the `DeriveMode` enum is removed.
Most callers will use `BonsaiDerived::derive`, or a default `derived_data_utils` implementation
that requires the derived data to be enabled and configured on the repo.
Backfillers can additionally use `derived_data_utils_for_backfill` which will use the
`backfilling` configuration in preference to the default configuration.
Reviewed By: ahornby
Differential Revision: D25246317
fbshipit-source-id: 352fe6509572409bc3338dd43d157f34c73b9eac
Summary:
Currently, data derivation for types that have options (currrently unode
version and blame filesize limit) take the value of the option from the
repository configuration.
This is a side-effect, and means it's not possible to have data derivation
types with different configs active in the same repository (e.g. to
server unodes v1 while backfilling unodes v2). To have data derivation
with different options, e.g. in tests, we must use `repo.dangerous_override`.
The first step to resolve this is to make the data derivation options a parameter.
Depending on the type of derived data, these options are passed into
`derive_from_parents` so that the right kind of derivation can happen.
The mapping is responsible for storing the options and providing it at the time
of derivation. In this diff it just gets it from the repository config, the same
as was done previously. In a future diff we will change this so that there
can be multiple configurations.
Reviewed By: krallin
Differential Revision: D25371967
fbshipit-source-id: 1cf4c06a4598fccbfa93367fc1f1c2fa00fd8235
Summary: Take the parameters to `derived_data_utils` and `derived_data_utils_unsafe` by reference.
Reviewed By: krallin
Differential Revision: D25371970
fbshipit-source-id: d260650c2398e33667e1bc5779fbabdff04f1f98
Summary:
The `BonsaiDerived` trait is split in two:
* The new `BonsaiDerivable` trait encapsulates the process of deriving the data, either
a single item from its parents, or a batch.
* The `BonsaiDerived` trait is used only as an entry point for deriving with the default
mapping and config.
This split will allow us to use `BonsaiDerivable` in batch backfilling with non-default
config, for example when backfilling a new version of a derived data type.
Reviewed By: krallin
Differential Revision: D25371964
fbshipit-source-id: 5874836bc06c18db306ada947a690658bf89723c
Summary: Looks like the permissions are different there. Let's glob it out.
Reviewed By: singhsrb
Differential Revision: D25507359
fbshipit-source-id: 6a5c19e41879798b829d9b6e79eba3009249c20c
Summary:
At the moment "hg pull -B bookmark" always fetches from infinitepush path even
if we do something like "hg pull -B master".
Let's fetch from infinitepush only if a bookmark matches scratch mather
Reviewed By: markbt
Differential Revision: D25460577
fbshipit-source-id: 6563dcd3423c6a7a70ea1c1f7acdaf5db5e21875
Summary: Two doctor tests don't pass on macOS, so disable them for now.
Reviewed By: genevievehelsel
Differential Revision: D25509814
fbshipit-source-id: c3fa92daefd4fda67335bdc66f56e35e94ae4e6a
Summary: Make the auth crate validate the user's certificate before returning it. This way we can catch invalid certs before trying to use them.
Reviewed By: sfilipco
Differential Revision: D25454687
fbshipit-source-id: ad253fb433310570c20f33dbd0d0bf11df21e966
Summary: Add a new module that can parse X.509 certificates and detect common issues (e.g., the certificate is missing, corrupt, or expired). This should allow us to provide better UX around certificate errors.
Reviewed By: sfilipco
Differential Revision: D25440548
fbshipit-source-id: b7785fd17fa85f812fd38de09e79420f4e256065
Summary: This makes it more flexible.
Reviewed By: kulshrax
Differential Revision: D24467604
fbshipit-source-id: 63023cf0dde2fb7eac592ac79008e4b7a62340c1
