Summary:
Passing shared_ptr by copy everywhere can be expensive as it forces an atomic
operation to be performed. Since the caller of the glob code can easily
guarantee that the data will outlive the globbing code, let's just pass
references/pointers to it so that only references and pointers are copied.
Reviewed By: genevievehelsel
Differential Revision: D31344889
fbshipit-source-id: cee797202470aa123381d9ee22e11780722f5b33
Summary: The endpoint can fail sometimes. Add retries to make it more reliable.
Reviewed By: yancouto
Differential Revision: D31407279
fbshipit-source-id: 1b05feedd65477aa0b9cd07be30217b4f2e6d1b2
Summary: This will be used by the next change.
Reviewed By: yancouto
Differential Revision: D31407281
fbshipit-source-id: 6e3e26a0fb4864d2076d2aafcb0f0ba919bfe2cc
Summary:
The `Response` type in edenapi has extra complexity about async streaming.
It is more useful for large data like file contents. For metadata like the
hash<->location translation, using a plain `Vec` is much simpler. Using
`Vec` also makes it easier to implement retry logic.
Reviewed By: yancouto
Differential Revision: D31407278
fbshipit-source-id: 9d6df225d183eaffe72f99cfd53ae3cd2987a518
Summary:
We concluded that ideally `sync -> block_on(async)` only happens in Python
wrappers, not pure Rust code. With that, `EdenApiBlocking` makes it easier
to write anti-pattern pure Rust code, and itself looks a bit repetitive.
All users of `EdenApiBlocking` have been migrated. So let's drop
`EdenApiBlocking` now.
Reviewed By: yancouto
Differential Revision: D31407284
fbshipit-source-id: 797506ccd07f413ac041dcd2ea07b3a3519d912f
Summary:
There are only a few places using EdenApiBlocking. The other endpoints are just
using explicit `block_on` constructs.
Let's migrate the endpoints off the EdenApiBlocking trait.
Reviewed By: yancouto
Differential Revision: D31407287
fbshipit-source-id: edeb16a1ed4f50cc01c75546df7362673f941d01
Summary:
EdenApiBlocking is going away. Drop another dependency on it.
This also adds proper Ctrl+C support for the command.
Reviewed By: yancouto
Differential Revision: D31407285
fbshipit-source-id: 6c5f40f2933fdd54db197ae6b93c5f240eda1c25
Summary:
This makes it possible to drop EdenApiBlocking. Since the actual lazy clone
logic is in the Python code, we might want to only keep one of the Rust or
Python lazy clone logic eventually.
Reviewed By: yancouto
Differential Revision: D31407286
fbshipit-source-id: 3685edbf6e4709aaf8190ba65035d6627ef83fe9
Summary:
`EdenApiBlocking` was there to provide some convenience for non-async code to
use edenapi. Now a lot of places actually use async Rust properly, and calling
async Rust from non-async Rust is not a great practice. So let's plan for
`EdenApiBlocking` removal. To unblock it, drop `EdenApiBlocking` usage in
revisionstore.
Reviewed By: yancouto
Differential Revision: D31407282
fbshipit-source-id: 70a6e9468606176cf5cf119418547e694e229ec4
Summary:
Add a config option `edenapi.max-retry-per-request` to control the maximum
retry count. Replace the hard-coded 10 with it.
Reviewed By: yancouto
Differential Revision: D31407280
fbshipit-source-id: 654c79601fe12007e6cbf7ac6a4110da420801dd
Summary: Add a general retry method for futures. Move clone and pull paths to use it.
Reviewed By: yancouto
Differential Revision: D31407283
fbshipit-source-id: 46ec2bd5bacdcd10ae5078ced8d12f2c8b9ed1ec
Summary:
The error is most likely a misconfiguration. Prompt the user to check the
message.
Differential Revision: D31383947
fbshipit-source-id: c58c2b026048266fc0a8c90f31928c97a2381258
Summary:
Using modulo on arbitrary integers to get random numbers [isn't correct](https://www.internalfb.com/diff/D31305392 (da13975a4f)?dst_version_fbid=311037904117090&transaction_fbid=550270779610744), as the distribution between numbers isn't fair (unless the size is a power of two).
This was raised on D31305392 (da13975a4f), but we decided to land that quickly to unblock builds before doing these changes.
I'm applying the changes suggested on D31305392 (da13975a4f). This is what this diff does:
- For all cases where we generate small numbers (up to 5), replace with call to `Gen::choose`, so `u32::arbitrary(g) % 3` becomes `g.choose(&[0, 1, 2]).unwrap()`.
- For generating numbers in range 0..=1, I instead replaced with generating a boolean, which gets rid of the `unreachable!` calls.
- I removed the code to generate numbers in range 0..=0.
- For generating larger numbers, I used `u64::arbitrary` instead, which should make things "less wrong".
Some things I assumed, but am happy to change before landing, just let me know:
- Theoretically we don't *need* to change the code for `% 2` and `% 4`, as the math checks out there. I changed it for consistency there, but am happy to change it back.
- Using boolean also wasn't suggested initially, I'm happy to change back.
Reviewed By: krallin
Differential Revision: D31379381
fbshipit-source-id: a0bac26ebabd32a6c65f717512de998ef5dc37c8
Summary:
`commit_lookup_pushrebase_history` is an endpoint that tries to traverse Pushrebase and Commit Sync mappings to find the commit's pushed version for a landed commit. But currently it traverses Commit Sync mapping blindly because it doesn't know the sync direction. This can (very rare) lead to an inaccurate result.
Let's use the `source_repo` column I've introduced in this diff stack and don't traverse in the wrong direction if we know it is wrong.
Reviewed By: StanislavGlebik
Differential Revision: D30975759
fbshipit-source-id: 9c5ecf059dcdebf0c91f0c5545f0c6e95610c2ec
Summary:
modernise tests by using moder newserver function
the function is shorter and it will also allow us to run the tests with Mononoke in the future
Reviewed By: yancouto
Differential Revision: D30773102
fbshipit-source-id: 994cbcfb2688aef3e96446e1cb021db72bc70c67
Summary:
Another test that was broken by quickcheck update. I notice it breaking a run of `hg_windows` [here](https://www.internalfb.com/intern/sandcastle/log/?instance_id=4503600123928895&step_id=4503604846860319&step_index=9&name=Run%20cargo%20test#479) (though it's not windows specific).
The problem is quickcheck started using NaN on floats, which surfaced some invalid code. In roundtrip tests, we need our struct to implement `Eq`, which f32 does not (since NaN != NaN). Workaround was implementing something that also considers NaN == NaN.
Question: Do I also fix the hg-server version of the test?
Reviewed By: krallin
Differential Revision: D31401277
fbshipit-source-id: b3eef1a3aef395a1194308788ec74f1bb5a33a42
Summary:
The backfiller should use the background session class so that we ensure data
is written to all blobstores in a multiplex.
Reviewed By: yancouto
Differential Revision: D31429168
fbshipit-source-id: 32c767dbef291771565f73cedf3cd01c1a3cce40
Summary:
When rederiving batches of commits, the batch derivation process must mark
these commits as derived so that rederivation will continue with the next
batch.
Reviewed By: yancouto
Differential Revision: D31429169
fbshipit-source-id: e9f6a84a0391ee8d72a0007f39e755410bfac724
Summary: Since we migrated all derived data types to use manager, we can now delete a bunch of code.
Reviewed By: krallin
Differential Revision: D31344999
fbshipit-source-id: db864bdc3ba0f95cb34be6e554d629d254f09608
Summary: The test output was previously incorrect due to wrong rounding from jq tool
Reviewed By: ahornby
Differential Revision: D31429221
fbshipit-source-id: 2979e393c6f6c1b52e41d732f155275166062bff
Summary:
fix the values in the test, the test was broken due to changes in jq
jq used to provide incorrect rounding.
Reviewed By: ahornby
Differential Revision: D31390426
fbshipit-source-id: ab4d7014109d23aa5b4fb95db4f485cea70e5b05
Summary: I'm going to reuse that derived type for unit-test in 2DService. And for that I need to make it library.
Reviewed By: HarveyHunt
Differential Revision: D31340518
fbshipit-source-id: 3960c0d3ae9a72e1fa6dc9afb170c0c708b3cdf8
Summary:
Writing to the LocalStore is purely the responsability of the
LocalStoreCachedBackingStore and not of the individual BackingStore. Thus, they
cannot assume that the root Tree is actually stored in it and should just
directly import it.
Reviewed By: chadaustin
Differential Revision: D31340206
fbshipit-source-id: 0f485ceb9fa71f7a7bdc8aaefaa850540075c88c
Summary:
Looking at Instruments, when issuing tons of glob queries to EdenFS, EdenFS
appears to be spending a very large amount of time adding tasks to the
UnboundedTaskExecutor. Since globs are expected to be fast, we can afford to
execute them inline, reducing this overhead and speeding up glob queries.
Reviewed By: chadaustin
Differential Revision: D31289485
fbshipit-source-id: 428fff9f5fea65073b2a061dc7070d63ae36d95d
Summary:
The globbing algorithm is recursive and returns its own glob results merged
with its children's glob results. The merging is done by simply copying the
children's glob result and returning it. What this means is that a single
GlobResult will be copied K times, with K being the recursion depth at which it
was created. This makes the total number of copies be O(K*N) with N the result
length.
Since we can simply avoid these copies by simply creating the GlobResult in a
shared vector, we can avoid the copies entirely at the expense of taking a
lock.
Reviewed By: chadaustin
Differential Revision: D31288036
fbshipit-source-id: ae8a98a01eab2ba7f23908d347d7a4ec199cdfab
Summary:
In the case where a child is already loaded, this allows the code to not
allocate a SemiFuture, which should benefit code that repeatly calls it on
already loaded inodes. One typical example is the globbing entry point that
needs to find the TreeInode where the glob needs to be applied on.
Reviewed By: chadaustin
Differential Revision: D31283398
fbshipit-source-id: 76f82d74f2a45ee2b3b9bf442d47c0a2262bced9
Summary:
One layer about getChildRecursive is getInode, let's make this one use an
ImmediateFuture too.
Reviewed By: chadaustin
Differential Revision: D31283397
fbshipit-source-id: 8bc524bea857d6ec5bc045d6e3383d38133c3b38
Summary:
This diff uses the helper proc macro added on the previous diff to simplify the code for dozens of api objects.
Over 1000 lines deleted :)
Examples of structs that couldn't be migrated:
- Wire structs that didn't rename the fields to numbers. (e.g. WireHistoryRequest) (would need some extra migration)
- enums (not currently supported by the proc macro)
- Wire structs that didn't map directly to their non-wire counterparts (e.g. WireSnapshotState)
I added some comments with possible future improvements, but didn't pursue them right now as they are significantly less useful than this diff itself, which covers most of the cases.
Reviewed By: ahornby
Differential Revision: D31057140
fbshipit-source-id: 88a867ba2cdfedf6a96a8ca3718508073822b962
Summary:
This diff implements a proc macro that derives a "wire" object from a simple api obj for edenapi. It is enough to cover most of the wire objects and simplify code a lot.
Differences from initial proof of concept:
- Renamed `default_wire` to `auto_wire`. I'm happy to rename again if preferred.
- Fields must have `#[id(X)]` notation.
- Fields with default values are not serialized.
- Arbitrary implementation also is derived
This diff only uses this proc_macro on one Api Object, in order to make reviewing it easier, focusing on the implementation of the macro. The next diff uses this macro in all possible objects, actually doing the bulk of the refactoring.
Reviewed By: ahornby
Differential Revision: D31054734
fbshipit-source-id: d6136faf84492983ca69461fe243e830021b2f66
Summary:
Plus a minor refactoring to use the io::IsTty trait in edenfs_client::status instead of calling into libc directly.
This commit reintroduces 218f06a4e648 after reversion in e9ef7c2142d0. Windows build broke because I accidentally left a stray `#[cfg(unix)]` above unrelated use line.
Differential Revision: D31248594
fbshipit-source-id: ddee62d9dc4d0b99d2e67fe9b757748ab501e030
Summary:
Setting EDENSCM_LOG can achieve the same effect. So let's remove the redundant
logic. This also avoids overhead constructing the strings when tracing is off.
Reviewed By: yancouto
Differential Revision: D31359046
fbshipit-source-id: db53cc16f1efcf6111535090a3eadec19b888329
Summary: Same as D30974102 (91c4748c5b) but for mercurial filenodes.
Reviewed By: markbt
Differential Revision: D31170597
fbshipit-source-id: fda62e251f9eb0e1b6b4aa950d93560b1ff81f67
Summary:
This fixes these tests (which are blocking the deployment):
```
FAILED eden/mononoke/mononoke_types:mononoke_types-unittest - content_metadata::test::content_metadata_blob_roundtrip
FAILED eden/mononoke/mononoke_types:mononoke_types-unittest - content_metadata::test::content_metadata_thrift_roundtrip
FAILED eden/mononoke/mononoke_types:mononoke_types-unittest - file_contents::test::file_contents_blob_roundtrip
FAILED eden/mononoke/mononoke_types:mononoke_types-unittest - file_contents::test::file_contents_thrift_roundtrip
```
They were broken since the update of quickcheck, as it now checks with big values for ints, which broke these tests as the code didn't do a good job of converting between u64/i64. This diff fixes that.
Usually, we just want to use `as` conversion everywhere. Thrift only stores i64. but it's fine to store a negative value that will correctly overflow back to the correct positive value of u64. In practice though, this shouldn't really happen, as these are sizes, and we're never getting anywhere near the u64 limit.
Using `TryInto` doesn't cause these overflows, but for just storage, it makes things work in less cases.
Reviewed By: krallin
Differential Revision: D31379001
fbshipit-source-id: feeb87a62148f97b3bd467e8c2ef2156c8e3329a
Summary: Establish a http connection to x2pagentd using a unix socket.
Reviewed By: mzr
Differential Revision: D31204332
fbshipit-source-id: d1f98dc0e4eae4fb918cd77a9571e1a2da7d7ed2
Summary:
If a file is replaced by a move or copy from another file, we should ignore
this copy info when computing the renames in a diff. If we do not, then we
will fail to include the deletion of the copy source in the case of a move, and
the copy information doesn't add anything anyway as the destination file will
just show as modified.
Reviewed By: mitrandir77
Differential Revision: D31180151
fbshipit-source-id: c89a8ae26a516fd958406bb967a587b3b6c36a48
Summary: Switch derivation of Git trees to use the `DerivedDataManager`.
Reviewed By: yancouto
Differential Revision: D31303798
fbshipit-source-id: 193f5d373a56a0d1099f49db76758227d15c3762
Summary: Add overrides for changesets, filenodes and bonsai_hg_mapping in the derived data manager.
Reviewed By: yancouto
Differential Revision: D31378456
fbshipit-source-id: b1faa543ca65fa041d2d0ddc908ea5fb950d023a
Summary:
Vendoring this patch: https://github.com/curl/curl/pull/7737 to curl-sys rust crate. On windows the hg client is using curl that's bundled with sys-curl. I need this patch to have unix domain sockets working in hg client on windows.
I had to manually vendor https://raw.githubusercontent.com/mzr/curl/57e7ec4dbe4dd2831de51f2644879387d2ea7b44/docs/INSTALL because reindeer didn't do it. IDK why.
oss-eden-{darwin,linux,windows}-getdeps fail with:
```
FAILED: eden/scm/lib/backingstore/CMakeFiles/rust_backingstore.cargo eden/scm/lib/backingstore/debug/libbackingstore.a eden/scm/lib/backingstore/release/libbackingstore.a
cd /data/sandcastle/temp/fbcode_builder_getdeps/shipit/eden/eden/scm/lib/backingstore && /data/sandcastle/temp/fbcode_builder_getdeps/installed/cmake-hQhVzQT-WzFKTeqXjLxo5lLi8IG4_MjX2-YRqptCUVs/bin/cmake -E remove -f /data/sandcastle/temp/fbcode_builder_getdeps/shipit/eden/eden/scm/lib/backingstore/Cargo.lock && /data/sandcastle/temp/fbcode_builder_getdeps/installed/cmake-hQhVzQT-WzFKTeqXjLxo5lLi8IG4_MjX2-YRqptCUVs/bin/cmake -E env CARGO_TARGET_DIR=/data/sandcastle/temp/fbcode_builder_getdeps/build/eden/eden/scm/lib/backingstore CARGO_HOME=/data/sandcastle/temp/fbcode_builder_getdeps/build/eden/_cargo_home cargo build --release -p backingstore --features fb
Blocking waiting for file lock on package cache
Blocking waiting for file lock on package cache
error: failed to calculate checksum of: /data/sandcastle/boxes/eden-trunk-hg-fbcode-fbsource/third-party/rust/vendor/curl-sys-0.4.45+curl-7.78.0/curl/docs/INSTALL
Caused by:
failed to open file `/data/sandcastle/boxes/eden-trunk-hg-fbcode-fbsource/third-party/rust/vendor/curl-sys-0.4.45+curl-7.78.0/curl/docs/INSTALL`
Caused by:
No such file or directory (os error 2)
```
Not idea how to fix this. Seems related to the fact that reindeer didn't vendor docs/INSTALL.
# EDIT:
# It might been caused by some bug in hg. now it's fine
# The failure in oss-eden-linux-getdeps looks unrelated (something with rocksdb)
Reviewed By: krallin
Differential Revision: D31370778
fbshipit-source-id: a1245f8cb6b58f5765e34c95dfd78325a8e6e457
Summary:
There were a few Wire objs that didn't have tests because they didn't implement Arbitrary. I added simple implementations for them.
I did it mostly for the request/response types, as they have the other types inside them.
Reviewed By: markbt
Differential Revision: D31019959
fbshipit-source-id: edf283fd79a40b794c89db79c026f1bcaf834a9f
Summary:
This diff adds snapshot tests for most eden api wire types, while at the same time making the testing code much smaller, including tests for "wire" and "serialize" roundtrips.
## Context:
The previous diff had added an easy way to add snapshot tests. This stack aims to simplify the wire protocol code needed to create/modify an endpoint. A good thing to do before that is to add snapshot tests to all wire types, so that if we change them in a refactor, we're confident they still work exactly the same. This will also be useful when a type is changed in the future.
## How this makes tests easier
- In order to create snapshot tests, we need example objects to test with. Luckily we already use a framework for generating example objects (quickcheck::Arbitrary), so the idea here is to use that to make snapshot tests as automatic as possible.
- At the same time, the "wire" and "serialize" roundtrip tests (which also used Arbitrary), can also be made more automatic.
Now, using a simple helper, `auto_wire_tests!(WireObjectName)`, it is possible to derive all three types of tests automatically. This makes the current code smaller, and safer as we now have the additional safety provided by snapshot tests.
## Observations
- Not all wire types had tests implemented for them (I assume because it was too much work doing so, and might have done that myself in the past), I only moved the ones that already had. I'll do another pass and add remaining objects on a following diff.
- There are a couple actual non-refactor changes. I'll add comments explaining those.
- quickcheck crate is using quite an old version. I tried updating but it snowballed into something much more complicated, so I kept using the old version. We'll need to get to it at some point, though.
Reviewed By: markbt
Differential Revision: D31019233
fbshipit-source-id: 30c4a90848d0a5dcaffb89b9a0cd1cebfe4ace55
Summary:
This diff adds a way to do snapshot tests in SC, by simply calling a macro. It uses that as an example on two wire objects to show it works.
Motivation:
- Using snapshot testing in wire object will make it super clear what changes are happening on the wire format, which is useful to prevent breakages. (see quip)
- I'm going with approach #3 from proposal, but both #2 and #3 would need snapshot testing.
How?
- I'm using [insta](https://docs.rs/insta/1.8.0/insta/), a rust crate for snapshot testing. Unfortunately it depends on some cargo-specific stuff that doesn't work with buck, but I was able to get it working without patching the package by bypassing the Cargo dependent stuff.
- Insta also provides some cargo exensions on top of it to compare snapshots, which we can't have, so I made it clear on the test errors (see test plan) how to update the snapshots (via flag).
Future
- On future diffs I'll add a snapshot test for all the wire objects, this just adds to a first few to show it works and set up the plumbing.
- Writing objects manually for snapshot tests for **every** wire object would be a large amount of code, and troublesome when adding new (specially big) wire objects. We already have wire objects implementing `Arbitrary`, so I think it should be possible to use this to generate simple snapshot tests on automatically generated objects. (It will need some extra work to make sure they are created consistently.)
Reviewed By: markbt
Differential Revision: D30958077
fbshipit-source-id: 3d8663e7897e5f6eb4b97c24f47b37ef2f138e5a
Summary:
This is fairly mechanical diff that finalizes split of Hash into ObjectId and Hash20.
More specifically this diff does two things:
* Replaces `Hash` with `Hash20`
* Removes alias `using Hash = Hash20`
Reviewed By: chadaustin
Differential Revision: D31324202
fbshipit-source-id: 780b6d2a422ddf6d0f3cfc91e3e70ad10ebaa8b4
Summary:
A spawner type is required for new thrift clients, specify the noop one for now.
This also requires regenerating the generated thrift libraries.
Reviewed By: yancouto
Differential Revision: D31338518
fbshipit-source-id: cbecf3ec6f9678918ca459c19f1cc160214fadfd
Summary:
We now use post-pull hook to mark commits as landed:
hooks.post-pull.marklanded=hg debugmarklanded
So there is no need to mark them inside the pull command. In fact, it seems
something is wrong (phases aren't invalidated properly?) so that the
pullcreatemarkers logic might actually hide commits during pull incorrectly:
commit 1e964a4302c03e5ae48e5b85b0fc0bf27f847b09
Author: metalog <metalog@example.com>
Date: Tue Sep 28 17:29:49 2021 +0000
pull
Parent: a4511a83cf862cc7216c15d83a3f4ff9d3b3241b
Transaction: pullcreatemarkers
RootId: 18d81a1531ecea65affe83c25804c790cac57c59
diff --git a/visibleheads b/visibleheads
index eb58137..da5f45a 100644
--- a/visibleheads
+++ b/visibleheads
@@ -1,16 +1,6 @@
v1
-e82de77adcc261cb306dafeb6cbe15f26f7de768
-91cf8c4b47e433acbe3f774e608eee42a3ad089d
8c071e5aa26d920f1b88c4b1cd10f6e946d4312a
536ff436cde4ec53d74c02ae2c5ed6f60609e01a
-e49b834a6ff9b61a95a743d22703dc6634f2918c
-c53b65542d4583ae82835144ac7f72dce7a6f335
1010312ed7ac683c5e97ad765cdbcb4927ddf62c
-a563a5f35d2df4c54ba1fe2401aa1cd929a218bf
-55c0c7483c6cd85114a9be587d11c87aaceaeff4
-74079567b677c5799b6c67855683ec346ebc3cce
33b5fd6055da2284cf5224f70e1bb9791ed87641
-ec221dff2393793f7b1e11ea9f9e9ea87b79da1f
-cc466614d43a6af24520b7a81653435f4a614fbb
2d7466be885e757d2d41630a3148fd31f5199ffa
-226c8136603dcbcc408133a2122e93fc045527fa
As we're here, document some other config options existed in the code.
Reviewed By: DurhamG
Differential Revision: D31295709
fbshipit-source-id: e26c728215a209ab5dfaee7a84daece8197a1cc4
Summary:
With the lookup processor now returning an ImmediateFuture, we can focus on its
caller, starting with getChildRecursive.
Reviewed By: chadaustin
Differential Revision: D31283396
fbshipit-source-id: 97abc57b9efe3540c5770aa952995c257e6eda4b
