Summary:
The broad goal here is to remove SegmentedChangelogBuilder.
Looking at Seeder dependencies here.
Reviewed By: StanislavGlebik
Differential Revision: D27202527
fbshipit-source-id: 164da1c4d202cc0f069f67963b71920dca9bcba5
Summary:
The original for this test was to desribe how the SegmentedChangelogBuilder was
to be used. We are removing SegmentedChangelogBuilder. This test goes away now.
Reviewed By: ahornby
Differential Revision: D27202521
fbshipit-source-id: e39a47411c61e8812d4081f6ee02323732369c1b
Summary:
This is part of removing SegmentedChangelogBuilder.
The Tailer constructor directly specifies the Mononoke requirements that is
needs to be provided in order to function.
Reviewed By: ahornby
Differential Revision: D27163312
fbshipit-source-id: 961066e2aa4e88c330841f7362b3ba17d0030638
Summary:
The `Request` builder's methods currently take `self` by value, which was intended to make it easy to create new `Request`s in a fluent style without assignment. Unfortunately, it turns out that this complicates situations where we need to modify a `Request` in-place.
The ideal solution would be to change `Request`'s builder methods to take `&mut self` instead [1]. I tried doing this, but unfortunately there are too many other parts of the current design that rely on the existing behavior in ways that are difficult to change (particularly around the creation of streaming and async requests).
As a workaround, this diff simply adds matching `fn set_X(&mut self, ...) -> &mut Self` methods for each builder method on `Request`. The existing consuming methods have been rewritten in terms of those methods to prevent duplication of the actual method contents.
Given that all of the consuming builders now contain essentially the same body, it seems like we could reduce the verbosity here by using a macro. Unfortunately, I'm not sufficiently experienced with writing nontrivial macros to come up with something quickly, but I do think that it would be a good idea to eventually use a macro here.
---
[1]: In fact, it turns out that [this is considered a better practice](https://github.com/rust-lang/api-guidelines/discussions/81) when designing builders in Rust -- as long as the terminal method of the builder chain returns the built struct by value, it is still possible to create a new instance without assignment. This does mean, however, that we cannot easily move things from the builder to the final struct without using tricks like `Option::take` and `mem::swap`, since the signature of the terminal method would be `(&mut self) -> Self`).
Reviewed By: andll
Differential Revision: D27256048
fbshipit-source-id: 14f770a87abc839d358e5ba211a096226d3e0dc6
Summary:
Similar to D27064825 (7eea44ce4e). There are 2 `hg status` calls and D27064825 (7eea44ce4e) only fixed one of them.
Sample failure:
--- test-doctor.t
+++ test-doctor.t.err
@@ -226,11 +226,11 @@
$ hg status
M A2
A A0
- A X
R A
R A1
? B
? C
+ ? X
? Y
? Z
Reviewed By: kulshrax
Differential Revision: D27249712
fbshipit-source-id: 0e628959c88218d6340f2597953850d654b12a8c
Summary:
Grouping stats by the shardmap can help to detect and root-cause issues.
This diffs adds a label to the `MysqlConnection` and Mononoke now will log counters by shardmap.
Reviewed By: StanislavGlebik
Differential Revision: D26994369
fbshipit-source-id: 0708a4b0dc3762f5f9152b83200173cd8b241abc
Summary:
This is a follow-up to D27093942 (59b8287c85). But D27093942 (59b8287c85) forgot one `read_log` spot, making it
possible to lose data.
Reviewed By: andll
Differential Revision: D27265233
fbshipit-source-id: ab1e73cdcfd3cc9da2e19000ee5fd5761977dc4a
Summary:
Turns out D27093942 (59b8287c85) was not a complete fix. It misses a `read_log`. Add a
test showing the problem.
Reviewed By: andll
Differential Revision: D27265232
fbshipit-source-id: 103e543abacbb2cbaf10aa3fe3d9693922d9daad
Summary:
There are a few abstractions in this that make it a little hard to
have different behavior between the download & upload paths.
This makes it hard to have different behavior for the two. It's a bit of a
problem right now in the sense that we end up doing things like sending a
Content-Length on a GET, but the reason I'm changing it is because I want to
chunk downloads and that requires having different logic in the upload &
download paths.
As part of doing this, I also moved a bunch of parameters away from
HttpLfsRemote and into HttpOptions, which makes the function signatures a
little more manageable.
Reviewed By: quark-zju
Differential Revision: D27191593
fbshipit-source-id: c332229bb3c5a4c1eaedb54dc12c3ddc19205050
Summary:
This will let us better load balance across our clients by having them send
requests that are a fixed size. Indeed, right now, load imbalanced between
our hosts seem to be largely attributable to the fact that some hosts are
responsible for a few 150MB+ blobs, whereas others have none of those.
The motivation here is to let clients query us by range. Then, we can just have
clients fetch e.g. ~40MB at a time, and route those to different hosts, thus
eliminating said load imbalance.
Reviewed By: StanislavGlebik
Differential Revision: D27188610
fbshipit-source-id: b9bbe18904cdd88ec7bab2b76e096fd3585c7b78
Summary:
Right now, this always returns the total size of the file instead of the bytes
we'll actually return. This is annoying because we use this to allocate
buffers in SCS.
Let's return the size of what will actually be returned.
Reviewed By: farnz
Differential Revision: D27188611
fbshipit-source-id: 0a05cc8ef6fb8e083dd0def405dfe1d0eecfd253
Summary:
I'd like to add support for Range queries in LFS. In HTTP, the range header
isn't start + size, it's start / end, so I'll need a slightly lower API. This
makes that possible.
Note: Range should have a private constructor, because otherwise we'll get
errors if start > end. Hence the `Range` / `RangeInner` here.
Reviewed By: StanislavGlebik
Differential Revision: D27187817
fbshipit-source-id: 20183d8b200fef6c16e45a66886ce8886d92e4f6
Summary: It's no longer in use, so remove it completely to simplify the code
Reviewed By: ahornby
Differential Revision: D27234699
fbshipit-source-id: ec26f4e283d48b05e19b951b8485ca6fa7751072
Summary:
Git client was updated and it started outputting a hint that breaks our tests. It has no meaning in tests so just quiet it.
```
➜ fbcode git init
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint: git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint: git branch -m <name>
Initialized empty Git repository in /data/users/mzr/fbsource/fbcode/.git/
```
Reviewed By: StanislavGlebik
Differential Revision: D27232853
fbshipit-source-id: 683ebebb36049adb758e7c26f843f12159a45301
Summary:
eden/scm/lib: default cpython_ext to python3
```
CI internally, try a PR on github once its landed.
Reviewed By: quark-zju
Differential Revision: D27237636
fbshipit-source-id: 1768f778d75b5d6c62dfd5641f911604a37d3163
Summary: Fix malformed module-level doc comment and rearrange import groups (based on the de facto Rust style conventions at FB).
Reviewed By: quark-zju
Differential Revision: D27243739
fbshipit-source-id: 21614d16e5fbfecd793939325ef52819db9a4cb8
Summary:
Hide private functions that might not have a stable interface guarantee in
`help`. For example, `help revset`:
Before:
Predicates
==========
The following predicates are supported:
"_all()"
All commits regardless of visibility
"_destrestack(SRC)"
restack destination for given single source
revision
"_localbranch"
"_localbranch(changectx)" localbranch commits
After:
Predicates
==========
The following predicates are supported:
"adds(pattern)"
Changesets that add a file matching pattern.
The pattern without explicit kind like "glob:" is expected to be
relative to the current directory and match against a file or a
directory.
Reviewed By: DurhamG
Differential Revision: D27130002
fbshipit-source-id: ee098489223c4033edeca2e5b6acab9475e63eb5
Summary:
Remove use of `#[async_trait]` in the new storage API, and eliminate the outer Future on the `fetch_stream` and `write_stream` methods, which was not used. The methods are now normal trait methods, which accept and return a stream.
It's possible we'll want to make these methods `async` again in the future, but until this this is more ergonomic, faster (one less layer of indirection), and avoids some of the type system limitations of `#[async_trait]`.
Reviewed By: andll
Differential Revision: D26695517
fbshipit-source-id: 2db60a3f37d594b0b9d1e1a2708532ef0ddaf585
Summary:
Introduce `FilterMapStore` combinator to support cases like LFS, where values of the same type may or may not be supported by a given store implementation.
Use `FilterMapStore` in Python bindings to prevent LFS files from being accidentally written to the non-LFS indexedlog.
With this change, the new storage API should be safe to use without corrupting local storage.
Reviewed By: kulshrax
Differential Revision: D26651254
fbshipit-source-id: 629cb43d85f43117a32b577777e13ff8fb801d57
Summary:
Introduce `HashMapStore`, a HashMap-based, generic, in-memory `ReadStore` / `WriteStore` implementation for testing purposes (and perhaps other uses in the future).
Introduce a minimal `KeyedValue` trait so that `HashMapStore` can determine the key to be associated with a written value in a generic way.
Reviewed By: kulshrax
Differential Revision: D26859235
fbshipit-source-id: 66032f072148796bde7a3a176fb2bb6707b95287
Summary:
Introduce `WriteError` type (similar to `FetchError` for reads) to support strongly-typed write errors. Eventually we'll probably want this type to carry the value that we tried to write (to allow write fallbacks, retries, etc) and perhaps add more enum variants (rejects, unavailable, etc).
Introduce `ReadWriteStore` trait for stores that support both reading and writing. This trait is automatically implemented for types which implement `ReadStore` and `WriteStore` with the same key and value types. This trait is intended to be used via the `BoxedRWStore` trait object. This type will be used in the Python bindings when `FallbackCache` is extended to implement `WriteStore` (in a future change).
Reviewed By: kulshrax
Differential Revision: D26646393
fbshipit-source-id: 96058cf1e826fdb6076a4162389829b3fe053686
Summary:
Introduce newfilestore class in pyrevisionstore, which constructs a newfilestore (BoxedReadStore for files) and a corresponding `ContentStore` which share the underlying IndexedLog object and EdenApi client.
Modify remotefilelog to construct ContentStore via this new class.
Currently, no methods are provided for the newfilestore - it is meant to be passed back into Rust code, where you may call it's Rust methods as shown in the `test_newstore` method (which will be removed in the future).
Currently the `util` module is made public for access from pyrevisionstore. In the future, this will be replaced in favor of a `NewFileStoreBuilder` which handles these concerns internally.
Reviewed By: DurhamG
Differential Revision: D26526331
fbshipit-source-id: c0f439fbee4c303db4a82171c866a3f3a5fc2324
Summary:
Split `FallbackStore` (which supports writing fallback fetches) into `FallbackCache` (which behaves the same) and `Fallback` (which doesn't support writing fallback fetches). This eliminates the `From<>` bound on `Fallback` which is nonsensical in some cases where we don't want writing.
Replace `write_store` and `write` fields with a single `Option<>` `write_store` in `FallbackCache`, which cleans up the code a bit.
Separate the "fallback value type", "preferred value type", "write value type", and "output value type" separately control the output value type independent of the preferred store.
Reviewed By: kulshrax
Differential Revision: D26526149
fbshipit-source-id: aa9f25f5efb8a9ddab03a7d86a7a007f24d156ce
Summary:
While clang has no issue compiling this code, gcc appears to choke on it,
failing to compile. This is unfortunate as this means we need to hardcode the
size of the serialized datastructure and validate it with a test.
Reviewed By: fanzeyi
Differential Revision: D27243075
fbshipit-source-id: 5cd59921bbd5d5be4dfb22789942eb022dac5bbe
Summary:
The problem appears when we have shared idmap moving ahead of the memory idmap
before we ever write to the memory idmap. In that case we would incorrectly
fetch last shared from the shared idmap. When that shared last entry is larger
than the cutoff we would try to assign ids starting from the shared entry. When
updating the IdDag it would assume that it has to insert all ids above the
cutoff but it would fail to resolve all ids exactly above the cutoff.
For example, MemIdMap is empty, cutoff is 5, shared idmap last entry is 7. We
asign 8-10 then the IdDag tries to search for 5-10 and fails to resolve 5.
This function was not updated after adding cutoff to OverlayIdMap in an earlier
diff.
Reviewed By: quark-zju
Differential Revision: D27248367
fbshipit-source-id: 97fc1efe8cdfb446c4571196dcef7c2db9a43330
Summary:
While Mononoke should support pushing large commits, it's not clear if we need
(or want) to support pushing a lot of commits. At the very least pushing lots of commits at the same
time can create problems with derivation, but there could be more places that might break.
Besides there's usually an easy way to work around that i.e. sqush commits or push in small batches.
If we ever need to import a lot of it we use tools like blobimport/repo_import,
so blocking pushing of lots of commits should be fine.
Reviewed By: farnz
Differential Revision: D27237798
fbshipit-source-id: 435e61110f145b06a177d6e492905fccd38d83da
Summary:
Add an `edenapi.logdir` config option, which, when set, will cause the EdenAPI client to write a JSON version of every request it sends to the specified directory. This is intended for use when reproducing user issues (so that the developer can inspect and replay the requests that were sent).
The JSON request files can be directly given to the `edenapi_cli` or converted to CBOR using the `make_req` tool for manual testing with `curl`.
The approach right now is a bit simplistic, in that we just write a JSON file whenever the EdenAPI client makes a request. I'm open to suggestions on how to improve this. (For example, if there were a way for us to always have a record of the last N requests, that would be helpful for debugging user issues.)
Reviewed By: DurhamG
Differential Revision: D27145093
fbshipit-source-id: 3834c2052b0c5efa05d1d209962953b29f545a3f
Summary: noticed these in passing while working on previous diff
Reviewed By: mitrandir77
Differential Revision: D27227682
fbshipit-source-id: e7858c81951b780722b0836ecf6ee72aeb1ffa09
Summary:
The latest git prints extra hints:
$ git init repo
+ hint: Using 'master' as the name for the initial branch. This default branch name
+ hint: is subject to change. To configure the initial branch name to use in all
+ hint: of your new repositories, which will suppress this warning, call:
+ hint:
+ hint: git config --global init.defaultBranch <name>
+ hint:
+ hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
+ hint: 'development'. The just-created branch can be renamed via this command:
+ hint:
+ hint: git branch -m <name>
Initialized empty Git repository in $TESTTMP/repo/.git/
Silent it to make the test pass.
Reviewed By: akushner
Differential Revision: D27235547
fbshipit-source-id: 6f7c8da3ac1e01ee9f57730e1586fa2053f9ca98