Summary: This is a nice to have check for correctness.
Reviewed By: DurhamG
Differential Revision: D31700272
fbshipit-source-id: 092a3fd463ae1a00e7c42645a1852054dadd8edf
Summary: This makes `all()` and `master_group()` aware of discontinuous segments.
Reviewed By: DurhamG
Differential Revision: D31734837
fbshipit-source-id: fe5ebd4930ef2de4540f37954558ed499072534b
Summary:
Use the added `Fold` feature from indexedlog. It's now possible to implement
`all_ids_in_groups` by using a "fold" function that tracks the covered spans.
Reviewed By: DurhamG
Differential Revision: D31734835
fbshipit-source-id: c9f10a8730563cb6803856519f4d3a7aec6d9dca
Summary:
Make `IdSet` and `Span` serializable. The `Id` type should really use
`#[serde(transparent)]` so it's just u64. However, adding
`#[serde(transparent)]` now on `Id` could be a breaking change.
So let's just ensure the `IdSet` use compact `Id` in serialization.
Reviewed By: DurhamG
Differential Revision: D31734840
fbshipit-source-id: e9f9ad7f3b1a5299cca2e8734c87bb5b03cf1f77
Summary:
Update FoldState on open, append so the fold state can be accessed.
This completes the Fold feature in indexedlog. See the added test for examples.
Reviewed By: DurhamG
Differential Revision: D31734836
fbshipit-source-id: 4683fdedd9ac07ff2a6a0b28a381b5ddbc3d17d8
Summary: This will be used by Log to incrementally calculate a fold function.
Reviewed By: DurhamG
Differential Revision: D31734834
fbshipit-source-id: 064bfa64d2faf8581a0c21f357cd9688c5d9ac23
Summary:
This allows us to cache the calculation of fold functions.
The on-disk and in-memory part are separate to handle various cases easier:
- Decide what to write to disk.
- Decide what to clone when `Log` is cloned.
Reviewed By: DurhamG
Differential Revision: D31734839
fbshipit-source-id: 45f8cffdf60bf2cb372f6d97ff9708a7a40ef134
Summary:
Prevously, a `Log` has a few index functions. The indexes are useful for
indexing individual items. This diff defines a new concept: "fold function".
It will be useful for accumulated information, such as the total count of
a special kind of entries.
This diff just defines the concept. Implementation will be done later.
Practically, this feature will be used to track the "assigned" segments in
the `dag` crate.
This feature cannot be easily built outside indexedlog because indexedlog
does not provide a way to iterate entries from an arbitrary offset. indexedlog
does not leak the raw offsets either. That makes incremental fold hard
to implement outside indexedlog.
Reviewed By: DurhamG
Differential Revision: D31734838
fbshipit-source-id: d48dca0da1c753d23cee725dc22a809d529c55db
Summary:
Update the `insert` API to take care of maintaining the `all_ids_in_groups`
state. Deserialization also goes through `insert` so that would work out of
box too.
Reviewed By: DurhamG
Differential Revision: D31734833
fbshipit-source-id: 53f37b951d7985436c4a90f80cc5f59155c7324d
Summary:
Partially address a PERF comment. The path would be useful in upcoming changes.
`cargo bench --bench spanset` now shows:
push_span (middle) 0.710 ms
Previously it was:
push_span (middle) 5.750 ms
Note the difference will be larger if the IdSet contains more spans.
Practically, we are thinking about 10k spans scale.
Reviewed By: DurhamG
Differential Revision: D31708352
fbshipit-source-id: cccbbab80fb997a50d5cc2536b076ad604f9b8fe
Summary:
Partially address a PERF comment. The path would be useful in upcoming changes.
`cargo bench --bench spanset` now shows:
push_span (high) 0.284 ms
Previously it was:
push_span (high) 5.557 ms
Reviewed By: DurhamG
Differential Revision: D31708351
fbshipit-source-id: b5c4238b82b711863779a6c9d24fa00d815f4ab2
Summary:
The operation will be used in a tight loop. This would demonstrate performance
differences.
`cargo bench --bench spanset` shows:
intersection 13.544 ms
union 8.055 ms
difference 6.438 ms
push_span (low) 0.173 ms
push_span (middle) 5.750 ms
push_span (middle, large) 5.883 ms
push_span (high) 5.557 ms
Note that we only have fast path for "low".
Reviewed By: DurhamG
Differential Revision: D31708350
fbshipit-source-id: c9b9da04f28cd8b5b4d05833812ff8d4fd0b94a2
Summary:
Make the IdDagStore implementation track what spans of Ids are taken.
This diff just defines the interface without actual implementation.
Where to track this information might be a bit not obvious. Here are some
choices and considerations:
- Tracking the spans in a new structure: More complex, since we need to ensure
IdMap, IdDag, and the new structure in sync. Requires non-trivial changes on
Mononoke.
- Tracking the spans in IdMap: IdMap does not depend on IdDag. So it feels
"natural". But it's non-trivial with IdMap when it's lazy, and it requires
non-trivial changes on Mononoke.
- Tracking the spans in IdDag: Easier, since IdDag is not async. Minimal
changes for Mononoke, since IdDag logic is fully shared. The only downside
is that IdMap will depend on IdDag to provide the spans. It feels cyclic,
but IdMap and IdDag are already somewhat tied together.
Reviewed By: DurhamG
Differential Revision: D31700271
fbshipit-source-id: a2461e0aafc15a03ffd82723b06ae4a77d95729b
Summary:
The actual implementation will be scattered in multiple components. Add a
single file with some basic explanation.
Reviewed By: DurhamG
Differential Revision: D31700274
fbshipit-source-id: 639332c55c29b4b1be8685f60685c9f696b293e8
Summary:
In order to execute the edenfs_privhelper, EdenFS needs to be a suid root
binary, but due to the large amount of dependency, it's hard to fully control
what runs before main and thus what executes with root privileges.
To reduce the attack surface, let's only make the edenfs_privhelper a suid root
binary. In tests, we still start EdenFS with sudo, and thus the
edenfs_privhelper will have the proper privileges.
Reviewed By: chadaustin
Differential Revision: D31870880
fbshipit-source-id: 3f06d22ecd4f3bc65e3e955c5a1de811b1cf328f
Summary:
Adds a command which prints out which directories were modified between
two diffs. "modified" is defined as "has an immediate child that has been added
or removed". This means just modifying a file will not cause it's parent
directory to be marked changed.
This is used for watchman to report what directories have changed during an eden
checkout.
Reviewed By: quark-zju
Differential Revision: D31698688
fbshipit-source-id: ef90573af3027306501673821ee40b691241b6e4
Summary:
Fix warnings in fbcode/eden/scm/lib/taggederror/src/lib.rs
$ buck build eden/scm/lib/taggederror:taggederror#check
```
warning: trailing semicolon in macro used in expression position
--> eden/scm/lib/taggederror/src/lib.rs:564:63
|
437 | bail!(IntentionalError(String::from("intentional_error")))
| ---------------------------------------------------------- in this macro invocation
...
564 | return std::result::Result::Err(anyhow::anyhow!($err));
| ^
|
= note: `#[warn(semicolon_in_expressions_from_macros)]` on by default
= warning: this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!
= note: for more information, see issue #79813 <https://github.com/rust-lang/rust/issues/79813>
= note: macro invocations at the end of a block are treated as expressions
= note: to ignore the value produced by the macro, add a semicolon after the invocation of `bail`
= note: this warning originates in the macro `bail` (in Nightly builds, run with -Z macro-backtrace for more info)
warning: trailing semicolon in macro used in expression position
--> eden/scm/lib/taggederror/src/lib.rs:497:95
|
442 | / bail!(
443 | | Category::Programming,
444 | | fault = Fault::Request,
445 | | TypeName("taggederror::FakeTypeNameForTesting"),
446 | | "intentional bail with {}",
447 | | "format params"
448 | | )
| |_____- in this macro invocation
...
497 | return std::result::Result::Err(anyhow::anyhow!($fmt, $($arg)*).with_metadata($meta));
| ^
|
= warning: this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!
= note: for more information, see issue #79813 <https://github.com/rust-lang/rust/issues/79813>
= note: macro invocations at the end of a block are treated as expressions
= note: to ignore the value produced by the macro, add a semicolon after the invocation of `bail`
= note: this warning originates in the macro `bail` (in Nightly builds, run with -Z macro-backtrace for more info)
warning: 2 warnings emitted
Downloaded 0/2 artifacts, 0.00 bytes, 100.0% cache miss (for updated rules)
Building: finished in 1.9 sec (100%) 20/20 jobs, 1/20 updated
Total time: 1.9 sec
More details at https://www.internalfb.com/intern/buck/build/92838d99-36e7-4c74-baea-9eb02b113cdc
BUILD SUCCEEDED
```
https://pxl.cl/1RVSM
Reviewed By: dtolnay
Differential Revision: D31933979
fbshipit-source-id: e43df45060d67acd714e6594e2682322e5f32515
Summary: This simply moves the conversion to folly::Future one level up.
Reviewed By: genevievehelsel
Differential Revision: D31746628
fbshipit-source-id: b64c17eefe949623f6b532d8d86aa703b65c99ad
Summary: This simply move the conversion to folly::Future to the caller.
Reviewed By: genevievehelsel
Differential Revision: D31746627
fbshipit-source-id: 944686bb5995d337e6cf66c13a3cd630ace74a6b
Summary: This moves the conversion to folly::Future up to the caller.
Reviewed By: genevievehelsel
Differential Revision: D31746630
fbshipit-source-id: 522169af5f7b978b88ebc8596558bd5c9b3c2482
Summary: This moves the conversion to folly::Future one caller up.
Reviewed By: genevievehelsel
Differential Revision: D31746633
fbshipit-source-id: 1af5f4f36c33da9248138ada112e277db239b0b1
Summary:
This merely moves the conversion from ImmediateFuture to Future in the
PrjfsChannel code.
Reviewed By: genevievehelsel
Differential Revision: D31746634
fbshipit-source-id: 9978c834e20f17c456322a32deaaa4c13e0cb270
Summary:
The timeout based notification mechanism appears to be at the root of a lot of
EdenFS crashes. It also doesn't detect all of the network failures and is being
replaced with a more robust mechanism which will handle these cases.
For now, let's just get rid of the timeout based mechanism.
Reviewed By: fanzeyi
Differential Revision: D31739683
fbshipit-source-id: d6baa03fe3579dc36c9f435bd16df39799e51ef9
Summary: This removes a bunch of conversion to a Future.
Reviewed By: genevievehelsel
Differential Revision: D31703355
fbshipit-source-id: 229b5fd55551a6d0b1346b8452fd1e57cb40e940
Summary: This merely pushes the conversion to a Future to the caller.
Reviewed By: genevievehelsel
Differential Revision: D31703354
fbshipit-source-id: 16446b4a03f18168af97134f08d374526c81adcd
Summary:
Add scuba logging for batch derivations.
In the request context scuba log, we log the batch start and end.
In the derived data scuba log, we log the full set of changeset ids for
the batch, however we do not log the serialized form of the mapped changeset
(this can be added, but it's likely to slow things down).
Reviewed By: StanislavGlebik
Differential Revision: D31902279
fbshipit-source-id: fda2bd608159a3d5e4c732a1cf76a02c0f7932cb
Summary: Per T102183795, we need to wrap all semaphores in task::spawn in order to reduce the risk of deadlocks.
Reviewed By: StanislavGlebik
Differential Revision: D31669763
fbshipit-source-id: 6c742d3794bc10bf80ef8e35a71ed465a7de637a
Summary: This allows pushing the transformation to a Future to the caller.
Reviewed By: chadaustin
Differential Revision: D31703356
fbshipit-source-id: 7e39b5d5862a7c8a2de646de515c5846a81c458a
Summary:
This simply bubbles up the ImmediateFuture to the caller, who already wanted an
ImmediateFuture, saving on conversion.
Reviewed By: kmancini
Differential Revision: D31703353
fbshipit-source-id: 698b3fe0e10a3fce919abd744a4ca3a4f2ff3878
Summary: This allows to remove some conversion from ImmediateFuture to folly::Future.
Reviewed By: chadaustin
Differential Revision: D31674457
fbshipit-source-id: 3e044776a3e9eb467131e309353f24774a13e549
Summary:
Add error contexts for derivation of batches in the derived data manager, and
for individual stacks within batches for blame_v2.
Reviewed By: StanislavGlebik
Differential Revision: D31898498
fbshipit-source-id: 0fec9672bfe55680a234624333ba7082de59bc69
Summary:
When splitting batches into linear stacks for parallel blame derivation, we
must also split on the existence of `copy_info`.
Blame derivation follows copy information, so if a commit in a batch modifies a
file, then a subsequent commit in the batch that copies from that file must be
placed in a separate stack so that it can access the blame data for the
copied-from file.
In theory it would be sufficient to check that copied-from sources aren't
modified by the previous commits in the stack, but this check isn't available
right now, so we fall back to just splitting on any copy info. We can add more
detailed splitting later if we need to improve performance further.
Reviewed By: StanislavGlebik
Differential Revision: D31898499
fbshipit-source-id: 85edb197231095de5ac17e07d5fab919277614ed
Summary: As discussed in previous diff, this removes constructor from hex string and adds ObjectId(fbstring) constructor
Reviewed By: chadaustin
Differential Revision: D31841234
fbshipit-source-id: c36ae315ad3a6eaecfd47889588c2bd18928aafb
Summary:
This mimics the one eden/fs/utils/Future.h behavior where all the Try are
unwrapped to simply return a tuple.
Reviewed By: kmancini
Differential Revision: D31668773
fbshipit-source-id: f4b9b6b2564cc1e2c43aa5b03bfb06048f0a39c1
Summary:
In some places, EdenFS uses folly::collectAll passing futures to it. In order
to replace it, we need an equivalent for ImmediateFuture.
The implementation relies on the fact that ImmediateFuture's constructor will
extract the value from a SemiFuture if it is ready, and thus the code doesn't
have to special case when all the passed in ImmediateFuture are immediate
values.
Reviewed By: chadaustin
Differential Revision: D31667897
fbshipit-source-id: ea680e1c74528d1eb34210cacb58efda8a26fee0
Summary: Hg mutation store is not in same mysql dbs as the rest of the metadata, revert connection caching for this for now.
Reviewed By: HarveyHunt
Differential Revision: D31898550
fbshipit-source-id: 2f1a237cc64f02f631d054208d0f1ddb62b41ac5
