Summary:
The broad goal is to get rid of SegmentedChangelogBuilder.
We will have a new constructor for Seeder, one that uses non segmented_changelog
dependencies as input.
Reviewed By: krallin
Differential Revision: D27202523
fbshipit-source-id: d420507502925d4440d5c3058efef0a4d2dbe895
Summary:
For tests that don't care about the bookmarks specifically, we want to use the
default bookmark name that we defined in BOOKMARK_NAME.
FWIW, it makes sense for this bookmark to be set by blobrepo even... at least
the fixtures. They set a bookmark and it makes sense for us to have a reference
to the bookmark that they set. Something to think about.
Reviewed By: krallin
Differential Revision: D27202522
fbshipit-source-id: 7615e4978dded491dd04ae44ce0b85134a252feb
Summary:
This gets rid of the odd builder for the Seeder.
We can get into design discussions with this one. What is struct and what is
function? For real structures that provide some behavior, I prefer to put
dependencies in owned data. Things that are part of the request go into
function parameters. In mononoke, RepositoryId is the common exception.
Anyway, IdMapVersion is part of the request for seeding. It is useful to have
that as a parameter when starting the seeder.
Reviewed By: krallin
Differential Revision: D27202528
fbshipit-source-id: a67b33493b20d2813fd0a144b9bb7f4510635ae8
Summary:
With the mix of external pager and progress suspension, the progress might
be enabled by accident:
# pager: disable forever
disable_progress(True)
# suspension
with progress.suspend()
...
# on __exit__, re-enables pager
Update the pager disabling logic be nested to avoid the potential issue.
Reviewed By: andll
Differential Revision: D27275016
fbshipit-source-id: 35ca7aef1890a981e6a1f0e5313b6a482ed43368
Summary:
Currently dropping progress bar panics if `__exit__` returns error
This happens, for example, when handling interrupts. Best course of actions just do not panic in this case
Reviewed By: sfilipco
Differential Revision: D27334897
fbshipit-source-id: c879fb14cfd4c16c0f9caede552129f8117defdc
Summary:
The progress total can be expensive to calculate for generatorset. Do not
calculate it. This will preserve laziness of certain sets (ex. generatorset
used by fastlog and average directory log).
Reviewed By: sfilipco
Differential Revision: D27327127
fbshipit-source-id: b0e3655e33b9e89ee2100941af18a769315f25bb
Summary:
Buffering the stream can provide suboptimal UX if the stream is slow.
Detect slow streams and avoid full buffering.
Reviewed By: sfilipco
Differential Revision: D27327128
fbshipit-source-id: a7b8037b7ba28fccc10661ffd15fd68f191d0048
Summary:
Remove use of dangerous_override from the repo client tests.
Previously this was used to override filestore config, so just use the existing
config override mechanism to set the filestore params this is generated from.
Reviewed By: ahornby
Differential Revision: D27169424
fbshipit-source-id: 7d17437f0e218d1cf19cc64d48e1efdd7012e927
Summary:
Remove use of dangerous_override in the pushrebase tests.
This was being used to ensure the bookmarks db in the test repo was shared with the mutable counters db. This is now directly possible by accessing the metadata database from the factory.
Reviewed By: ahornby
Differential Revision: D27169435
fbshipit-source-id: 1412231bdd9214bc869a3bfa7f63bf6c14db6836
Summary:
Remove uses of dangerous_override from derived data tests.
These were being used to override the derived data config or the lease object.
Extend the factory to allow customization of the config or the lease object,
and use that instead.
Reviewed By: StanislavGlebik
Differential Revision: D27169438
fbshipit-source-id: e8d0be248391d02bb054e19fdb9a90005db09c84
Summary:
Remove uses of dangerous_override in the commit rewriting tests.
Previously, the test was using dangerous_override to replace the bookmarks
attributes with ones that share a backing database with a `SqlMutableCounters`
instance.
With TestRepoFactory, all attributes share one metadata database. Obtain that
from the factory and use it to initialize the `SqlMutableCounters` instance.
Reviewed By: StanislavGlebik
Differential Revision: D27169429
fbshipit-source-id: 3c1b285db38a96deca7029d37e6692cb49356d31
Summary:
Remove dangerous overrides used in the tests in the bookmarks crate.
The test was using dangerous overrides to change the blobstore, which is now
supported directly by the test repo factory.
It was also using dangerous overrides to override the bookmark update log to
reset its database so it looks like the log is empty. Instead, clear out the
bookmark update log database as part of the test.
Reviewed By: StanislavGlebik
Differential Revision: D27169426
fbshipit-source-id: 64e1e89e31f62dcb585741ea728ebbe45f60fd38
Summary: This has been superseded by `test_repo_factory::TestRepoFactory`.
Reviewed By: StanislavGlebik
Differential Revision: D27169434
fbshipit-source-id: 97fcd400c5e3c6e8f86c9acfc5e979909e2eda31
Summary: Use the test factory for the remaining existing tests.
Reviewed By: StanislavGlebik
Differential Revision: D27169443
fbshipit-source-id: 00d62d7794b66f5d3b053e8079f09f2532d757e7
Summary: Use the test factory for existing bookmarks and pushrebase tests.
Reviewed By: StanislavGlebik
Differential Revision: D27169430
fbshipit-source-id: 65a7c87bc37cfa2b3b42873bc733cec177d8c1b0
Summary: Use the test factory for test fixtures that are used in many existing tests.
Reviewed By: StanislavGlebik
Differential Revision: D27169432
fbshipit-source-id: bb3cbfa95b330cf6572d1009c507271a7b008dec
Summary: Use the test factory for existing lfs_server tests.
Reviewed By: StanislavGlebik
Differential Revision: D27169427
fbshipit-source-id: 004867780ad6a41c3b17963006a7d3b0e5b82113
Summary: Use the test factory for existing commit_rewriting tests.
Reviewed By: StanislavGlebik
Differential Revision: D27169442
fbshipit-source-id: df2447b2b6423d172e684d7e702752ad717a2a4b
Summary: Use the test factory for existing repo_client and repo_import tests.
Reviewed By: StanislavGlebik
Differential Revision: D27169425
fbshipit-source-id: 2d0c34f129447232cec8faee42056d83613de179
Summary: Use the test factory for existing hooks and mapping tests.
Reviewed By: StanislavGlebik
Differential Revision: D27169436
fbshipit-source-id: f814e462bb2244e117c83bd355042016f5bfde8e
Summary: Use the test factory for existing mononoke_api tests.
Reviewed By: StanislavGlebik
Differential Revision: D27169444
fbshipit-source-id: d940bfa494dfe89fdb891d5248b73ba764f74faf
Summary: Use the test factory for existing derived data tests.
Reviewed By: farnz
Differential Revision: D27169433
fbshipit-source-id: b9b57a97f73aeb639162c359ab60c6fd92da14a8
Summary:
Create a factory that can be used to build repositories in tests.
The test repo factory will be kept in a separate crate to the production repo factory, so that it can depend on a smaller set of dependencies: just those needed for in-memory test repos. This should eventually help make compilation speeds faster for tests.
A notable difference between the test repos produced by this factory and the ones produced by `blobrepo_factory` is that the new repos share the in-memory metadata database. This is closer to what we use in production, and in a couple of places it is relied upon and existing tests must use `dangerous_override` to make it happen.
Reviewed By: ahornby
Differential Revision: D27169441
fbshipit-source-id: 82541a2ae71746f5e3b1a2a8a19c46bf29dd261c
Summary:
Convert `BlobRepo` to a `facet::container`. This will allow it to be built
from an appropriate facet factory.
This only changes the definition of the structure: we still use
`blobrepo_factory` to construct it. The main difference is in the types
of the attributes, which change from `Arc<dyn Trait>` to
`Arc<dyn Trait + Send + Sync + 'static`>, specified by the `ArcTrait` alias
generated by the `#[facet::facet]` macro.
Reviewed By: StanislavGlebik
Differential Revision: D27169437
fbshipit-source-id: 3496b6ee2f0d1e72a36c9e9eb9bd3d0bb7beba8b
Summary: To prepare for making `RepoBlobstore` a facet, convert it to a newtype wrapper.
Reviewed By: ahornby
Differential Revision: D27169439
fbshipit-source-id: ceefe307e962c03c3b89be660b5b6c18d79acf3e
Summary: We have support for backup-repo-id, but tw blobimport doesn't have id and have source repo name to use. Let's add support similar to other repo-id/source-repo-id etc.
Reviewed By: StanislavGlebik
Differential Revision: D27325583
fbshipit-source-id: 44b5ec7f99005355b8eaa4c066cb7168ec858049
Summary:
I'm trying to track down an issue in SQLBlob construction, where it takes over 3 wall-clock seconds to construct.
This is indicative of a missing `blocking` annotation somewhere; to make it easier to add the correct annotation, switch construction to modern futures and `async`
Reviewed By: krallin
Differential Revision: D27275801
fbshipit-source-id: 2b516b4eca7143e4be17c50c6542a9da601d6ac6
Summary:
The pack key needs to be removed using unlink() after target keys are pointed to it so that old unused packs can be GCed. e.g.
packer run 1: keys A and B are packed to P1
packer run 2: keys A and C are packed to P2
packer run 3: keys B and D are packed to P3
If we don't unlink P1 during run1 then GC can't collect the old pack after run 3 as the key will keep it live
Reviewed By: farnz
Differential Revision: D27188108
fbshipit-source-id: bbe22a011abbf85370a8c548413401ffb54b16b6
Summary:
Refactor `scmstore::types` into separate `file` and `tree` modules and introduce a new `StoreTree` type to represent trees in the scmstore API.
Introduce a minimal `StoreTree` type. Importantly, for now this type does not provide any methods for deserializing the tree manifest and inspecting it's contents. This functionality won't be too hard to implement, though - it'll require some revisions to the `manifest-tree` crate and / or moving the `StoreTree` and `StoreFile` types to `revisionstore_types`.
Reviewed By: kulshrax
Differential Revision: D27310878
fbshipit-source-id: 712330fba87f33c49587fa895efea3601ce377af
Summary:
The revset is not optimized for the (to be migrated to segments) revlog backend.
Optimize it.
Reviewed By: jteosw
Differential Revision: D27317708
fbshipit-source-id: cec9d6aad0f6c30c69a931898f8e1cc7c904b3f8
Summary:
We've seen occasional index timeouts on inserts in MySQL. This is very
reminiscent of D19158550 (fef360b284).
aida points out this seems to have started to happen (occasionally) recently.
That would make sense: we used to insert one-by-one so we wouldn't have ordering
issues (because bug), but we also recently started inserting in batches again (because bugfix).
There's little reason to expect we couldn't run into the same bug there as well.
So let's give filenodes & copydata the same sorting treatment we give paths.
Worst case, this does nothing. Best case, it fixes the issue.
Reviewed By: StanislavGlebik
Differential Revision: D27301588
fbshipit-source-id: 2b24ddd68e1a1c4e31fe33e03efcef47dad3657d
Summary: Previously, accessing both repo.filelog and repo.manifestlog in a test debug command, under buck builds only, would cause a "memcache singleton leaked" error to be printed after the command completes (see https://www.internalfb.com/intern/qa/87181/runtime-warnings-about-memcacheconnectionmanagerim ). There are still some unanswered questions about this issue, as noted in the QA post, but this fixes the main issue at hand.
Reviewed By: sfilipco
Differential Revision: D27297498
fbshipit-source-id: e19665333bae9f91e1c3c6db370962a3aea2727d
Summary:
Rename "newstore" to "scmstore"
The three main options I'm considering are "edenstore", "scmstore", and "hgstore". They all describe the project sufficiently well, in my opinion. "edenstore" might be a stretch, as I have no reals plans for Mononoke to use it, while "hgstore" might be a bit limited, as integration with EdenFS is a core goal, and it'll be used by EdenFS to fetch remote data via EdenApi, not just Mercurial's local storage. I feel like "scmstore" rolls off the tongue the easiest, particularly vs. "hgstore" (both "H.G. Store" and "Mercurial Store"), and is a bit easier to type than "edenstore" (which has "ede" all with the same finger). I don't have a strong opinion on this matter, so If you'd like a different name I'm open to suggestions.
Speak now or forever hold your peace.
Reviewed By: andll
Differential Revision: D27180332
fbshipit-source-id: 19e6972ea0f6527e671792845dcfd339cf1ab767
Summary:
Introduce a new `StoreFile` type for the `revisionstore` crate. This is an enum of `File`, `LfsPointer`, and `RedactedFile`, which represent the different cases the `Entry` type's `content` might actually represent. The `File` variant also handles stripping the copy header, if present, and stores both the `copied_from`, raw unstripped `content`, and stripped `content`, though only the latter can be accessed through the public API right now. Ideally, we'll move copy information into the history API and never actually need to make it public here.
Conversions are provided from `Entry` and EdenApi's `FileEntry` type (which also supports redaction and LFS pointers, but as errors instead of first-class values).
Modify the output type used in the Python bindings to use this new `StoreFile` type.
Currently, the `LfsPointer` variant is unused. This will be used later when we add first-class LFS support.
Reviewed By: kulshrax
Differential Revision: D26862710
fbshipit-source-id: 8326921f3ee43bf2e253847d5735c61f5a50bfa6
Summary: Adds `ExtStoredPolicy` support to the EdenApi and IndexedLog ReadStore implementations. This flag controls how LfsPointers found during a file query are handled (either returned as a result, or treated as "not found").
Reviewed By: kulshrax
Differential Revision: D27171814
fbshipit-source-id: 14dda47f32184c3ee703fbc77106885ca4d3ea27
Summary:
Add an unlink operation to BlobstoreWithLink for use from packblob in the next diff.
The reason an unlink operation is needed is so that pack keys don't keep old packed data live longer than it should be. e.g.
Before the packer runs, keys A, B, C, D each point to content A, B, C, D
packer run 1: keys A and B are packed to P1. Keys A, B, P1 now all point to pack P1
GC run here can remove content A and B
packer run 2: keys A and C are packed to P2. Keys A, C, P2 now all point to P2.
GC run here can remove content C
packer run 3: keys B and D are packed to P3. Keys B, D, P3 now all point to P3.
GC run here can remove content B and D.
At this point, keys A, C point to pack P2, keys B, D point to pack P3.
If we don't unlink pack P1 during packer run 1 then GC can't collect the old pack after run 3 as the key P1 will keep it live; if we unlink key P1 after pointing keys A and B to point at P1, then when keys A and B stop pointing to pack P1, there will be no further references.
The unlink method is on the BlobstoreWithLink trait which most Mononoke code does not reference and therefore can't easily accidentally call.
Its modelled on posix unlink(2), it will error if the key being unlinked does not exist.
Reviewed By: farnz
Differential Revision: D27188109
fbshipit-source-id: 628698dd5894895e2aff5cf7db979d32e1bc2b3e
Summary: For batch deriviation of filenodes we can first dervie non-root filenodes and write them into db and after that write all root-filenodes for that batch of commits.
Reviewed By: StanislavGlebik
Differential Revision: D27044302
fbshipit-source-id: 2a2eb8e336c787c4210192500b4d4a2a0663a29c
Summary:
This used to be very useful in the early stages, as a way to manually test the
code, now that the NFS procotol is pretty well defined and tests are actually
running, this has outlived its usefulness, let's simply remove the code.
Reviewed By: kmancini
Differential Revision: D27194039
fbshipit-source-id: af86edd9f438448209a7d14ba66c9b54d90a9594
Summary:
When I wrote the NFS code, I used `std::move` a bit too much, on datastructure
where moving them is equivalent to copying them. Instead, we can simply use
references, which makes the code shorter, and thus more efficient.
Reviewed By: kmancini
Differential Revision: D27172574
fbshipit-source-id: d9f06bf3f519e3539cf5cd0a0c4e4a49ef8009a8
Summary:
This should have been added in D27243075 (5a150e125a) but I forgot to run `hg add` and it
was thus not added...
Reviewed By: fanzeyi
Differential Revision: D27279169
fbshipit-source-id: 69807cc05fef33f51b2a491b66c2e8aeb7136deb
Summary:
Direct reads mean we cut the Manifold server out of the loop for bigger reads, and is an enabler for Hedwig and other ways to distribute a big blob between tasks in a region.
Weak consistency means that we don't always check the master server for metadata for a blob, which should speed up reads especially in remote regions; it depends on direct reads to function, as it allows us to grab the read plan from Memcache and then go directly to the backend storage (ZippyDB, Everstore) instead of via the Manifold servers
Reviewed By: StanislavGlebik
Differential Revision: D27264690
fbshipit-source-id: 78fe77dfa4f85b5bf657b64e56577f4e00af0b94
Summary:
Make the `RequestCreationEventListeners::new_request` event listener take an `&mut Request` instead of an `&mut RequestContext` as a parameter.
In the existing code (particularly in the `hg-http` crate), this event listener is used to configure the `RequestContext` for reporting progress. This diff just generalizes this idea, allowing the listener to modify the entire `Request`.
This is useful when we need to hijack request creation in `hg-http` to do Mercurial-specific configuration. The specific use case here is to disable TLS certificate checking when the global `--insecure` flag is set.
(Note that `http-client` itself is application-agnostic, so Mercurial specific configuration should not happen in this crate. This is why `hg-http` exists at all.)
Reviewed By: quark-zju
Differential Revision: D27242947
fbshipit-source-id: 019e19037642fe24acaa8c2917d446b91e7bcb26
Summary: Add `verify_tls_cert` and `verify_tls_host` settings to `http-client::Request` that are equivalent to [`CURLOPT_SSL_VERIFYPEER`](https://curl.se/libcurl/c/CURLOPT_SSL_VERIFYPEER.html) and [`CURLOPT_SSL_VERIFYHOST`](https://curl.se/libcurl/c/CURLOPT_SSL_VERIFYHOST.html). These will be used to allow skipping cert validation with the `--insecure` flag.
Reviewed By: quark-zju, sfilipco
Differential Revision: D27242946
fbshipit-source-id: cfa0fe800d0d132ca10ec0203bfd20b53c68b814
Summary:
The broad goal here is to remove SegmentedChangelogBuilder.
Looking at Seeder dependencies here.
Reviewed By: StanislavGlebik
Differential Revision: D27202527
fbshipit-source-id: 164da1c4d202cc0f069f67963b71920dca9bcba5
Summary:
The original for this test was to desribe how the SegmentedChangelogBuilder was
to be used. We are removing SegmentedChangelogBuilder. This test goes away now.
Reviewed By: ahornby
Differential Revision: D27202521
fbshipit-source-id: e39a47411c61e8812d4081f6ee02323732369c1b
Summary:
This is part of removing SegmentedChangelogBuilder.
The Tailer constructor directly specifies the Mononoke requirements that is
needs to be provided in order to function.
Reviewed By: ahornby
Differential Revision: D27163312
fbshipit-source-id: 961066e2aa4e88c330841f7362b3ba17d0030638
Summary:
The `Request` builder's methods currently take `self` by value, which was intended to make it easy to create new `Request`s in a fluent style without assignment. Unfortunately, it turns out that this complicates situations where we need to modify a `Request` in-place.
The ideal solution would be to change `Request`'s builder methods to take `&mut self` instead [1]. I tried doing this, but unfortunately there are too many other parts of the current design that rely on the existing behavior in ways that are difficult to change (particularly around the creation of streaming and async requests).
As a workaround, this diff simply adds matching `fn set_X(&mut self, ...) -> &mut Self` methods for each builder method on `Request`. The existing consuming methods have been rewritten in terms of those methods to prevent duplication of the actual method contents.
Given that all of the consuming builders now contain essentially the same body, it seems like we could reduce the verbosity here by using a macro. Unfortunately, I'm not sufficiently experienced with writing nontrivial macros to come up with something quickly, but I do think that it would be a good idea to eventually use a macro here.
---
[1]: In fact, it turns out that [this is considered a better practice](https://github.com/rust-lang/api-guidelines/discussions/81) when designing builders in Rust -- as long as the terminal method of the builder chain returns the built struct by value, it is still possible to create a new instance without assignment. This does mean, however, that we cannot easily move things from the builder to the final struct without using tricks like `Option::take` and `mem::swap`, since the signature of the terminal method would be `(&mut self) -> Self`).
Reviewed By: andll
Differential Revision: D27256048
fbshipit-source-id: 14f770a87abc839d358e5ba211a096226d3e0dc6
Summary:
Similar to D27064825 (7eea44ce4e). There are 2 `hg status` calls and D27064825 (7eea44ce4e) only fixed one of them.
Sample failure:
--- test-doctor.t
+++ test-doctor.t.err
@@ -226,11 +226,11 @@
$ hg status
M A2
A A0
- A X
R A
R A1
? B
? C
+ ? X
? Y
? Z
Reviewed By: kulshrax
Differential Revision: D27249712
fbshipit-source-id: 0e628959c88218d6340f2597953850d654b12a8c
Summary:
Grouping stats by the shardmap can help to detect and root-cause issues.
This diffs adds a label to the `MysqlConnection` and Mononoke now will log counters by shardmap.
Reviewed By: StanislavGlebik
Differential Revision: D26994369
fbshipit-source-id: 0708a4b0dc3762f5f9152b83200173cd8b241abc
Summary:
This is a follow-up to D27093942 (59b8287c85). But D27093942 (59b8287c85) forgot one `read_log` spot, making it
possible to lose data.
Reviewed By: andll
Differential Revision: D27265233
fbshipit-source-id: ab1e73cdcfd3cc9da2e19000ee5fd5761977dc4a
Summary:
Turns out D27093942 (59b8287c85) was not a complete fix. It misses a `read_log`. Add a
test showing the problem.
Reviewed By: andll
Differential Revision: D27265232
fbshipit-source-id: 103e543abacbb2cbaf10aa3fe3d9693922d9daad
Summary:
There are a few abstractions in this that make it a little hard to
have different behavior between the download & upload paths.
This makes it hard to have different behavior for the two. It's a bit of a
problem right now in the sense that we end up doing things like sending a
Content-Length on a GET, but the reason I'm changing it is because I want to
chunk downloads and that requires having different logic in the upload &
download paths.
As part of doing this, I also moved a bunch of parameters away from
HttpLfsRemote and into HttpOptions, which makes the function signatures a
little more manageable.
Reviewed By: quark-zju
Differential Revision: D27191593
fbshipit-source-id: c332229bb3c5a4c1eaedb54dc12c3ddc19205050
Summary:
This will let us better load balance across our clients by having them send
requests that are a fixed size. Indeed, right now, load imbalanced between
our hosts seem to be largely attributable to the fact that some hosts are
responsible for a few 150MB+ blobs, whereas others have none of those.
The motivation here is to let clients query us by range. Then, we can just have
clients fetch e.g. ~40MB at a time, and route those to different hosts, thus
eliminating said load imbalance.
Reviewed By: StanislavGlebik
Differential Revision: D27188610
fbshipit-source-id: b9bbe18904cdd88ec7bab2b76e096fd3585c7b78
Summary:
Right now, this always returns the total size of the file instead of the bytes
we'll actually return. This is annoying because we use this to allocate
buffers in SCS.
Let's return the size of what will actually be returned.
Reviewed By: farnz
Differential Revision: D27188611
fbshipit-source-id: 0a05cc8ef6fb8e083dd0def405dfe1d0eecfd253
Summary:
I'd like to add support for Range queries in LFS. In HTTP, the range header
isn't start + size, it's start / end, so I'll need a slightly lower API. This
makes that possible.
Note: Range should have a private constructor, because otherwise we'll get
errors if start > end. Hence the `Range` / `RangeInner` here.
Reviewed By: StanislavGlebik
Differential Revision: D27187817
fbshipit-source-id: 20183d8b200fef6c16e45a66886ce8886d92e4f6
Summary: It's no longer in use, so remove it completely to simplify the code
Reviewed By: ahornby
Differential Revision: D27234699
fbshipit-source-id: ec26f4e283d48b05e19b951b8485ca6fa7751072
Summary:
Git client was updated and it started outputting a hint that breaks our tests. It has no meaning in tests so just quiet it.
```
➜ fbcode git init
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint: git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint: git branch -m <name>
Initialized empty Git repository in /data/users/mzr/fbsource/fbcode/.git/
```
Reviewed By: StanislavGlebik
Differential Revision: D27232853
fbshipit-source-id: 683ebebb36049adb758e7c26f843f12159a45301
Summary:
eden/scm/lib: default cpython_ext to python3
```
CI internally, try a PR on github once its landed.
Reviewed By: quark-zju
Differential Revision: D27237636
fbshipit-source-id: 1768f778d75b5d6c62dfd5641f911604a37d3163
Summary: Fix malformed module-level doc comment and rearrange import groups (based on the de facto Rust style conventions at FB).
Reviewed By: quark-zju
Differential Revision: D27243739
fbshipit-source-id: 21614d16e5fbfecd793939325ef52819db9a4cb8
Summary:
Hide private functions that might not have a stable interface guarantee in
`help`. For example, `help revset`:
Before:
Predicates
==========
The following predicates are supported:
"_all()"
All commits regardless of visibility
"_destrestack(SRC)"
restack destination for given single source
revision
"_localbranch"
"_localbranch(changectx)" localbranch commits
After:
Predicates
==========
The following predicates are supported:
"adds(pattern)"
Changesets that add a file matching pattern.
The pattern without explicit kind like "glob:" is expected to be
relative to the current directory and match against a file or a
directory.
Reviewed By: DurhamG
Differential Revision: D27130002
fbshipit-source-id: ee098489223c4033edeca2e5b6acab9475e63eb5
Summary:
Remove use of `#[async_trait]` in the new storage API, and eliminate the outer Future on the `fetch_stream` and `write_stream` methods, which was not used. The methods are now normal trait methods, which accept and return a stream.
It's possible we'll want to make these methods `async` again in the future, but until this this is more ergonomic, faster (one less layer of indirection), and avoids some of the type system limitations of `#[async_trait]`.
Reviewed By: andll
Differential Revision: D26695517
fbshipit-source-id: 2db60a3f37d594b0b9d1e1a2708532ef0ddaf585
Summary:
Introduce `FilterMapStore` combinator to support cases like LFS, where values of the same type may or may not be supported by a given store implementation.
Use `FilterMapStore` in Python bindings to prevent LFS files from being accidentally written to the non-LFS indexedlog.
With this change, the new storage API should be safe to use without corrupting local storage.
Reviewed By: kulshrax
Differential Revision: D26651254
fbshipit-source-id: 629cb43d85f43117a32b577777e13ff8fb801d57
Summary:
Introduce `HashMapStore`, a HashMap-based, generic, in-memory `ReadStore` / `WriteStore` implementation for testing purposes (and perhaps other uses in the future).
Introduce a minimal `KeyedValue` trait so that `HashMapStore` can determine the key to be associated with a written value in a generic way.
Reviewed By: kulshrax
Differential Revision: D26859235
fbshipit-source-id: 66032f072148796bde7a3a176fb2bb6707b95287
Summary:
Introduce `WriteError` type (similar to `FetchError` for reads) to support strongly-typed write errors. Eventually we'll probably want this type to carry the value that we tried to write (to allow write fallbacks, retries, etc) and perhaps add more enum variants (rejects, unavailable, etc).
Introduce `ReadWriteStore` trait for stores that support both reading and writing. This trait is automatically implemented for types which implement `ReadStore` and `WriteStore` with the same key and value types. This trait is intended to be used via the `BoxedRWStore` trait object. This type will be used in the Python bindings when `FallbackCache` is extended to implement `WriteStore` (in a future change).
Reviewed By: kulshrax
Differential Revision: D26646393
fbshipit-source-id: 96058cf1e826fdb6076a4162389829b3fe053686
Summary:
Introduce newfilestore class in pyrevisionstore, which constructs a newfilestore (BoxedReadStore for files) and a corresponding `ContentStore` which share the underlying IndexedLog object and EdenApi client.
Modify remotefilelog to construct ContentStore via this new class.
Currently, no methods are provided for the newfilestore - it is meant to be passed back into Rust code, where you may call it's Rust methods as shown in the `test_newstore` method (which will be removed in the future).
Currently the `util` module is made public for access from pyrevisionstore. In the future, this will be replaced in favor of a `NewFileStoreBuilder` which handles these concerns internally.
Reviewed By: DurhamG
Differential Revision: D26526331
fbshipit-source-id: c0f439fbee4c303db4a82171c866a3f3a5fc2324
Summary:
Split `FallbackStore` (which supports writing fallback fetches) into `FallbackCache` (which behaves the same) and `Fallback` (which doesn't support writing fallback fetches). This eliminates the `From<>` bound on `Fallback` which is nonsensical in some cases where we don't want writing.
Replace `write_store` and `write` fields with a single `Option<>` `write_store` in `FallbackCache`, which cleans up the code a bit.
Separate the "fallback value type", "preferred value type", "write value type", and "output value type" separately control the output value type independent of the preferred store.
Reviewed By: kulshrax
Differential Revision: D26526149
fbshipit-source-id: aa9f25f5efb8a9ddab03a7d86a7a007f24d156ce
Summary:
While clang has no issue compiling this code, gcc appears to choke on it,
failing to compile. This is unfortunate as this means we need to hardcode the
size of the serialized datastructure and validate it with a test.
Reviewed By: fanzeyi
Differential Revision: D27243075
fbshipit-source-id: 5cd59921bbd5d5be4dfb22789942eb022dac5bbe
Summary:
The problem appears when we have shared idmap moving ahead of the memory idmap
before we ever write to the memory idmap. In that case we would incorrectly
fetch last shared from the shared idmap. When that shared last entry is larger
than the cutoff we would try to assign ids starting from the shared entry. When
updating the IdDag it would assume that it has to insert all ids above the
cutoff but it would fail to resolve all ids exactly above the cutoff.
For example, MemIdMap is empty, cutoff is 5, shared idmap last entry is 7. We
asign 8-10 then the IdDag tries to search for 5-10 and fails to resolve 5.
This function was not updated after adding cutoff to OverlayIdMap in an earlier
diff.
Reviewed By: quark-zju
Differential Revision: D27248367
fbshipit-source-id: 97fc1efe8cdfb446c4571196dcef7c2db9a43330
Summary:
While Mononoke should support pushing large commits, it's not clear if we need
(or want) to support pushing a lot of commits. At the very least pushing lots of commits at the same
time can create problems with derivation, but there could be more places that might break.
Besides there's usually an easy way to work around that i.e. sqush commits or push in small batches.
If we ever need to import a lot of it we use tools like blobimport/repo_import,
so blocking pushing of lots of commits should be fine.
Reviewed By: farnz
Differential Revision: D27237798
fbshipit-source-id: 435e61110f145b06a177d6e492905fccd38d83da
Summary:
Add an `edenapi.logdir` config option, which, when set, will cause the EdenAPI client to write a JSON version of every request it sends to the specified directory. This is intended for use when reproducing user issues (so that the developer can inspect and replay the requests that were sent).
The JSON request files can be directly given to the `edenapi_cli` or converted to CBOR using the `make_req` tool for manual testing with `curl`.
The approach right now is a bit simplistic, in that we just write a JSON file whenever the EdenAPI client makes a request. I'm open to suggestions on how to improve this. (For example, if there were a way for us to always have a record of the last N requests, that would be helpful for debugging user issues.)
Reviewed By: DurhamG
Differential Revision: D27145093
fbshipit-source-id: 3834c2052b0c5efa05d1d209962953b29f545a3f
Summary: noticed these in passing while working on previous diff
Reviewed By: mitrandir77
Differential Revision: D27227682
fbshipit-source-id: e7858c81951b780722b0836ecf6ee72aeb1ffa09
Summary:
The latest git prints extra hints:
$ git init repo
+ hint: Using 'master' as the name for the initial branch. This default branch name
+ hint: is subject to change. To configure the initial branch name to use in all
+ hint: of your new repositories, which will suppress this warning, call:
+ hint:
+ hint: git config --global init.defaultBranch <name>
+ hint:
+ hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
+ hint: 'development'. The just-created branch can be renamed via this command:
+ hint:
+ hint: git branch -m <name>
Initialized empty Git repository in $TESTTMP/repo/.git/
Silent it to make the test pass.
Reviewed By: akushner
Differential Revision: D27235547
fbshipit-source-id: 6f7c8da3ac1e01ee9f57730e1586fa2053f9ca98
Summary:
We need to bump SCS counters expressing Mononoke's QPS. They will look something like:
`requests:mononoke:oregon:carolina` for requests coming from proxygen in prn and mononoke in frc.
CSLB expects regions' full names.
We're getting src region from proxygen as a header.
Reviewed By: krallin
Differential Revision: D27082868
fbshipit-source-id: 12accb8a9df5cf6a80c2c281d2f61ac1e68176d1
Summary:
Add `repo_derived_data`. This is a new struct that encapsulates derived data
configuration and lease operations, and will be used in the facet-based
construction of repositories.
Reviewed By: ahornby
Differential Revision: D27169431
fbshipit-source-id: dee7c032deb93db8934736c111ba7238a6aaf935
Summary:
Add `repo_identity`. This is a new struct that encapsulates repository
identity and will be used in the facet-based construction of repositories.
Reviewed By: ahornby
Differential Revision: D27169445
fbshipit-source-id: 02a435bba54a633190c6d2e4316e86726aecfdf0
Summary:
In preparation for making `BlobRepo` buildable by facet factories, restore
`BlobRepo` members that had been converted to `TypeMap` attributes back into
real members.
This re-introduces some dependencies that were previously removed, but this
will be cleaned up when crates no longer have to depend on BlobRepo directly,
just the traits they are interested in.
Reviewed By: ahornby
Differential Revision: D27169422
fbshipit-source-id: 14354e6d984dfdd2be5c169f527e5f998f00db1e
Summary:
Resolve a circular dependency whereby `BlobRepo` needs to depend on
`Arc<dyn SegmentedChangelog>`, but the segmented changelog implementation
depends on `BlobRepo`, by moving the trait definition to its own crate.
Reviewed By: sfilipco
Differential Revision: D27169423
fbshipit-source-id: 5bf7c632607dc8baba40d7a9d65e96e265d58496
Summary:
a workspace could be renamed from a different machine
educate users that they need to switch to a valid workspace
Reviewed By: markbt
Differential Revision: D27155544
fbshipit-source-id: eed066f2f3e6ebf99732499fdb355f8aebb4c1df
Summary:
this could happen if it has been removed from a different machine
In this case, we should educate the users about the '--force' option
Reviewed By: markbt
Differential Revision: D27155423
fbshipit-source-id: 41cc3ac769dfd4145031fef687e8069d0ef8f4c9
Summary:
provide better ux for switching workspace if the current has been renamed from another machine
sometimes users rename a workspace but it's unclear for them how to switch other machines to the new one
we should educate them about '--force' option
Reviewed By: markbt
Differential Revision: D27117194
fbshipit-source-id: faef1cf9ce64f054f715ef3683a133d3088ddc72
Summary:
We've been seeing flaky test output due to python2 and python3 stdout being different
Bring getdeps builds unambiguously onto python3.
Reviewed By: mitrandir77
Differential Revision: D27190200
fbshipit-source-id: e53327fc3293a0ccdf88d8a199b66eddb4b9b8e4
Summary: There were a few remaining python2 parts to remove and the error case from hgcommand.run() had wrong encoding
Reviewed By: mitrandir77
Differential Revision: D27194046
fbshipit-source-id: 8c154204991e5a587f70ae4b18c73c4f005880ab
Summary:
Before I use a new command line argument I need to add it first.
This one will be set to the top level smc tier which is mononoke, mononoke.experimental, etc.
It will be used to bump SCS counters consumed by CSLB.
I don't want this to be optional because I don't want people to accidentally turn off publishing SCS counters representing QPS. On the other hand I don't want people who run it locally to pollute prod counters (in the script that sets this up locally its set to some dommy tier).
Reviewed By: krallin
Differential Revision: D27190020
fbshipit-source-id: 472011f0584012b036efdd015565cda3b3c029e9
Summary: For the prefetch-profile feature, added a matching enable option to go with the existing disable option. Enable temporarily enables the prefetch-profile feature for cases where a user has already used disable.
Reviewed By: kmancini
Differential Revision: D27191711
fbshipit-source-id: cf9cb7d4054e9fcee66bdb5f205562ec79c2757e
Summary:
When copying files from Finder to EdenFS, Finder appears to be issuing a couple
of getxattr calls to EdenFS, however having these being denied by osxfuse
causes the copy to fail, resulting in a subpar experience.
Reviewed By: fanzeyi
Differential Revision: D27137328
fbshipit-source-id: 66bcc2e316d832385da05f0768da9f82ab36a374
Summary:
This is unused, and the Thrift file mentions that it should be fine to remove
in July 2020. It's now March 2021, time to kill it.
Reviewed By: chadaustin
Differential Revision: D26852134
fbshipit-source-id: 2872185edd834f889b78802210071d16b881e14c
Summary: It's best to test for a regular file by comparing dtype_t, not mode_t directly.
Reviewed By: chadaustin
Differential Revision: D26851735
fbshipit-source-id: 62d3833a53748ea9b45794ec6390d158e888c241
Summary: It doesn't have to be optional
Reviewed By: johansglock
Differential Revision: D27188336
fbshipit-source-id: 796778484aa4c1a455a356b0f412e2abe5522ed0
Summary:
Like it says in the title. See the previous diff for context. This is a little
more complex than I'd like, and IMO largely had to do with the fact that we
have one method for mapping Bonsais AND hg, and that's a bit messy. I think
this is worth refactoring, but in the interest of keeping the diffs small, this
right now just ports the current API to use rendezvous under the hood.
Reviewed By: StanislavGlebik
Differential Revision: D27010315
fbshipit-source-id: 9bef893dd37db244b0b91f5de4fa79d0fdc47228
Summary:
This introduces a basic building block for query batching. I called this
rendezvous, since it's about multiple queries meeting up in the same place :)
There are a few (somewhat conflicting) goals this tries to satisfy, so let's go
over them:
1), we'd like to reduce the total number of queries made by batch jobs. For
example, group hg bonsai lookups made by the walker. Those jobs are
characterized by the fact that they have a lot of queries to make, all the
time. Here's an example: https://fburl.com/ods/zuiep7yh.
2), we'd like to reduce the overall number of connections held to MySQL by
our tasks. The main way we achieve this is by reducing the maximum number of
concurrent queries. Indeed, a high total number of queries doesn't necessarily
result in a lot of connections as long as they're not concurrent, because we
can reuse connections. On the other hand, if you dispatch 100 concurrent
queries, that _does_ use 100 connections. This is something that applies to
batch jobs due to their query volume, but also to "interactive" jobs like
Mononoke Server or SCS, just not all the time. Here's an example:
https://fburl.com/ods/o6gp07qp (you can see the query count is overall low, but
sometimes spikes substantially).
2.1) It's also worth noting that concurrent queries are often the result of
many clients wanting the same data, so deduplication is also useful here.
3), we also don't want to impact the latency of interactive jobs when they
need to a little query here or there (i.e. it's largely fine if our jobs all
hold a few connections to MySQL and use them somewhat consistently).
4), we'd like this to make it easier to do batching right. For example, if
you have 100 Bonsais to map to hg, you should be able to just map and call
`future::try_join_all` and have that do the right thing.
5), we don't want "bad" queries to affect other queries negatively. One
example would be the occasional queries we make to Bonsai <-> Hg mapping in
`known` for thousands (if not more) of rows.
6), we want this to be easy to incorporate into the codebase.
So, how do we try to address all of this? Here's how:
- We ... do batching, and we deduplicate requests in a batch. This is the
easier bit and should address #1, #2 and #2.1, #4.
- However, batching is conditional. We notably don't batch very large requests
with the rest (addresses #5). We also don't batch small queries all the time:
we only batch if we are observing a throughput of queries that suggests we
can find some benefit in batching (this targets #3).
- Finally, we have some utilities for common cases like having to group by repo
id (this is `MultiRendezVous`), and this is all configurable via tunables
(and the default is to not do anything).
Reviewed By: StanislavGlebik
Differential Revision: D27010317
fbshipit-source-id: 4a2397255f9785c6722c02e4d419438fd0aafa07
Summary:
Generic structs can't be used as facets, which prevents `RepoBonsaiSvnrevMapping` being
used as a facet. Currently we only use it with `Arc<dyn BonsaiSvnrevMapping>` anyway,
so make that official by removing the generic parameter.
Reviewed By: ahornby
Differential Revision: D27169420
fbshipit-source-id: 908b1555341652e72adad087bc0b77565cd75b9d
Summary:
It's really annoying that we're not throwing away the cause here, because when
you get a crash now you get zero context. This fixes that. I'd rather not say
how long it took me to find this line...
Reviewed By: StanislavGlebik
Differential Revision: D27168808
fbshipit-source-id: ac3312c2f3b328f08929be499c45c4462cb159a3
Summary:
EdenFS doesn't register itself against the portmap client, and on some system
where it is not started, it appears to not work reliably, crashing EdenFS early
at startup. For now, let's only build it when services need to be registered.
Reviewed By: genevievehelsel
Differential Revision: D27162906
fbshipit-source-id: cc2a8a588a756e54253da31f9bc00fbe4e5312d9
Summary:
In a bunch of places, the code assumes that an EdenMount is associated with a
Fuse channel. With NFS, that's no longer the case, thus let's make sure to
check the return value of mount->getFuseChannel(). In the case where it will
make sense to have something for NFS, I've either added an EDEN_BUG, or a TODO,
so we can come back to it later.
Reviewed By: chadaustin
Differential Revision: D26836431
fbshipit-source-id: c061b8f20199e5af3139a5003827f184f6eac8d4
Summary:
In the case where EdenFS would fail to initialize very early, the optional
server would be empty, and thus trying to dereference it would lead to EdenFS
crashing with no good error message. Let's simply test for this.
Reviewed By: genevievehelsel
Differential Revision: D27162907
fbshipit-source-id: a078b9995a94e5a86cf2893cbebfe7f5ca6d064e
Summary:
Currently we can only limit which users are allowed to move a bookmark by a
regex. We also want to allow specifying a hipster group.
Reviewed By: krallin
Differential Revision: D27156690
fbshipit-source-id: 99a5678a82f4c34ed2e57625361ba7cdb08ed839
Summary:
See the previous diff for context. Disable the Rust progress for external
pager.
Reviewed By: kulshrax
Differential Revision: D27149241
fbshipit-source-id: 4260a8be55bbfa648d8910f021195e9d11bdab73
Summary:
When testing "disable_progress" with chg (next diff) I found it was not
effective because there are 2 separate IO structs. The one we disable
from Python is different from the one the Rust progress thread uses.
I traced it down here. Since the Python IOs are just wrappers of
Rust IOs in the chg use-case. There is no need to recreate an IO
struct.
The "creating IO" struct is still useful, for things like "-t.py" testing where
the output needs to be captured into different Python variables per different
commands.
Reviewed By: DurhamG
Differential Revision: D27149243
fbshipit-source-id: 6e27adcc9f48b21fc24fba120be8c4a8fef1f909
Summary:
In some cases (ex. using an external pager). The IO states are changed outside
the IO struct's control. Ideally we should implement the external pager logic
on IO too but for now let's just add an API so the Python external pager logic
can disable progress output after starting an external pager.
Reviewed By: kulshrax
Differential Revision: D27149242
fbshipit-source-id: ff51fc153d3cc211cfa8ef697923d36f7c0f0d9b
Summary:
Detecting prod on Windows wasn't working because we used a posix path.
Let's add the Windows equivalent.
Also moves us to use the new hostcaps crate.
Reviewed By: chadaustin
Differential Revision: D27126497
fbshipit-source-id: 4035012fb7701378fb6e2e902c0efcd54ef42ea9
Summary:
We've been seeing issues where repositories end up with incorrect
dynamic configuration since there's a window of time after they're cloned where
they don't have %include /etc/mercurial/.../repo.rc and therefore generate an
incorrect dynamicconfig which gets used for 15 minutes until we regen the
dynamicconfig.
Let's change hg clone to write the %include as part of the initial hgrc, so we
remove that window of time and the repo will always be correctly configured.
Reviewed By: quark-zju
Differential Revision: D27093772
fbshipit-source-id: a9ca0ec54e06549546d532d1c49a80d49981decf
Summary:
I'd like to add support for event handlers in the Rust ServiceRouter client (I
need this in order to inject CATs in calls made by the SMC client). To do so, I
need to be able to instantiate a `ContextStack`, and to do so, I need a static
c-string representing the service name & function name, which is what this diff
does.
Note that we actually do the same thing for Rust servers already.
#forcetdhashing
Reviewed By: farnz
Differential Revision: D27088187
fbshipit-source-id: be2ad541d5123b31f0dab73da16b35dbfd308d6f
Summary:
The NFS readdir turns out to be pretty similar to the FUSE one, with a couple
of differences. For one, it only populates the directory entry name, it also
puts a limit on the total size of the serialized result, including all the
NFS/XDR overhead.
It is not specified if the . and .. entries need to be returned, but since the
NFS spec is usually pretty explicit about these and makes it clear that this is
for the most part a client burden, I didn't add these. I may have to revisit
this later when I get to manually browse a repository.
Since the READDIR RPC doesn't populate any filehandle, the client will have to
issue a LOOKUP RPC for each entries, potentially leading to some
inefficiencies. A future diff will implement the READDIRPLUS to fix these.
Reviewed By: chadaustin
Differential Revision: D26802310
fbshipit-source-id: b821b57021d0c2dca33427975b1acd665173bc5c
Summary:
This simplifies a handful of tests and will make writing the READDIR RPC a bit
less magic when computing the amount of memory needed per entry.
Reviewed By: chadaustin
Differential Revision: D26802312
fbshipit-source-id: fc66cb68f721ed34c8f9879cdda2cd8db6ed8daa
Summary: This merely adds the types for the READDIR RPC.
Reviewed By: chadaustin
Differential Revision: D26802313
fbshipit-source-id: 634ff9b3f97dc4dba56d225c1fb9eae0a94c02d5
Summary:
Looking at the spec, READDIRPLUS appears to be more complex to implement than
READDIR, for now, let's force the use of READDIR. Future changes will have to
implement READDIRPLUS as that will likely be a perf improvement.
Reviewed By: chadaustin
Differential Revision: D26802311
fbshipit-source-id: cb784d74507e6c2c2ba4dc0aebe69cfcd69db40b
Summary:
This type is very specific to Fuse, let's make it obvious. The readdir method
has also been renamed as it is also very specific to Fuse.
Reviewed By: chadaustin
Differential Revision: D26802309
fbshipit-source-id: c2acdfd1c0006935c59b685fcda729e1bef88928
Summary:
This test verifies that the issue we had previously with assign_ids does not
creep up again.
Reviewed By: quark-zju
Differential Revision: D27105741
fbshipit-source-id: 49b385b2026b599c92c406331a2299931a2eae46
Summary: Update the logs so that it's more clear what is going on.
Reviewed By: quark-zju
Differential Revision: D27145099
fbshipit-source-id: 11ec7b467157d07dd41893dc82f251a1c555365f
Summary:
We are also going to make update the IdMapVersionStore before we start writing
the IdMap. This means that if we crash while writing the IdMap, future runs
don't try to use the same IdMapVersion that we used previously.
Reviewed By: quark-zju
Differential Revision: D27145097
fbshipit-source-id: b911e2dca32d0fe8ae0aead3de75373dd2f936c4
Summary:
We are going to build the iddag before starting to write the idmap.
This means if the iddag fails to build for whatever reason we would not have
written a potentially useless idmap.
Reviewed By: quark-zju
Differential Revision: D27145098
fbshipit-source-id: c9045abea2a1f5a8b96c524d546776fdc693b56a
Summary:
`update::build` is only used by the Seeder. The steps in this function are not
isolated enough from the seeder to have a separate function. The seeder has the
role of builing it's own type of StartState. It is also the only process that
deals with the IdMapVersionStore. The seeder is particular enough that it makes
sense to inline it's build order.
Reviewed By: quark-zju
Differential Revision: D27099265
fbshipit-source-id: f86b8d7d4637a5f2582e70fc58b60c2041b93548
Summary:
The most important invariant for IdDag is that parent nodes have ids that are
smaller than child nodes. We had a couple of issues that resulted in failing
this invariant so we are adding these extra checks. They will help us diagnose
issues faster and proctect protect production data against faulty updates.
Reviewed By: quark-zju
Differential Revision: D27092204
fbshipit-source-id: 1f052b290a494e267fac2f551ba51582baa67973
Summary: Shadowing can end up being more confusing.
Reviewed By: quark-zju
Differential Revision: D27143481
fbshipit-source-id: 0a1913d8952fe913cc7596b9aea84df2d62cc3fe
Summary:
Check that head has a dag id assignment after finishing the process. This was
done at a later point but it is better to group it with assignment process so
that we have a clear source of the error.
Reviewed By: quark-zju
Differential Revision: D27143482
fbshipit-source-id: 2a94cee70142967b4f8d57df43dfcc339a0b4f2e
Summary: Move around code, similar to other data types.
Reviewed By: StanislavGlebik
Differential Revision: D27044301
fbshipit-source-id: 6c1c104533592733e95c3976717c5ac484218c6f
Summary:
Like it says in the title. We do the same thing (with the sampling rates) in
repo client.
Reviewed By: mitrandir77
Differential Revision: D27156569
fbshipit-source-id: ffaec7e27b454650263e82fd6d18f25c1bbf88eb
Summary:
We use srselect on hg hosts to figure out which Mononoke servers hgcli can connect to.
We were getting BrokenPipe errors most likely from srselect.
`E0318 03:51:02.558708 679 [tk] eden/mononoke/server/repo_listener/src/connection_acceptor.rs:201] connection_acceptor error: Failed to handle connection to [2401:db00:12:90c3:face:0:361:0]:57594: Failed to handle_connection: Failed to handle_http: Failed to serve_connection: error shutting down connection: Broken pipe (os error 32): Broken pipe (os error 32)
`
Tbh it isn't much different than `ErrorKind::NotConnected` because we'are shutting the connection down anyway and it doesn't matter whether client hung up or is already dead.
Reviewed By: krallin
Differential Revision: D27155123
fbshipit-source-id: 96bb2b268f116a20f16605eb04c867c9ad047b1f
Summary: We have new config fields available that can specify default compression level, let's read and use them.
Reviewed By: StanislavGlebik
Differential Revision: D27127455
fbshipit-source-id: 27935fd58da5f1150c9caf56d9601c37f2ae3581
Summary:
In regular xcode this was warning and being ignores. Not the working is handled as an error.
This diff is only a workaround so we wont get those errors .
```
eden/scm/edenscm/mercurial/cext/osutil.c:745:49: error: passing 'const char *' to parameter of type 'char *' discards qualifiers [-Werror,-Wincompatible-pointer-types-discards-qualifiers]
ret = _listdir_batch(path, pathlen, keepstat, skip, &fallback);
^~~~
eden/scm/edenscm/mercurial/cext/osutil.c:586:11: note: passing argument to parameter 'skip' here
char* skip,
```
Reviewed By: mzlee
Differential Revision: D27136440
fbshipit-source-id: 00d61fd00e3ed8e23643ea69b5a82dbeb5e742ce
Summary:
With `sl -r OBSOLETED` the intention is see the obsoleted stack instead of just
a single commit. So filter the "::heads" with "- public()", not "& draft()".
The goal is to deprecate `--hidden`. See the linked post for more context.
Reviewed By: DurhamG
Differential Revision: D27093425
fbshipit-source-id: 76e9650a809c1d94da2341e2aca31d349487610d
Summary:
When creating the .hg directory, Mercurial issues a SYMLINK RPC, thus let's
support it.
Reviewed By: kmancini
Differential Revision: D26785005
fbshipit-source-id: a760d55e6117cc3725444c604e3e4036f4a317b2
Summary:
For now, this simply clone a repo with NFS, and nothing else, more of the
protocol needs implementing to support reading directories, files, etc.
Reviewed By: kmancini
Differential Revision: D26266144
fbshipit-source-id: e379e12126162f41d8d166bb53652e1e501de2e9
Summary:
Better Git<->Bonsai conflict reporting.
It now prints the conflicting mapping, so one can see if there is a 2xbcs_id -> same git_sha1 or 2xgit_sha1 -> same bcs_id conflict. This is useful when trying to sort out why this issue appeared in the first place.
Reviewed By: ahornby, krallin
Differential Revision: D27058044
fbshipit-source-id: 9db7a210c1b0be3e0e90aa78b561293d6cf29c26
Summary:
Allow to use a custom accumulator inside gitimport so we selectively can decide what to save.
This was triggered mainly because we run out of memory due to the large BonsaiChangesets always collected even when not needed earlier.
Reviewed By: krallin
Differential Revision: D27117686
fbshipit-source-id: 99ce33562e76470f91ff8c0c46391bd513801afa
Summary:
This is mostly just copying /usr/include/linux/fuse.h from my devserver and
updating some flags in FuseChannel to display the new flags.
Reviewed By: chadaustin
Differential Revision: D27144667
fbshipit-source-id: 4854c6edd4c793ca707db26fecd11e2a3e9d7b75
Summary:
Segmented Changelog distinguishes commits into two groups: MASTER and
NON_MASTER. The MASTER group is assumed to big and special attention is payed
to it. Algorithms optimize for efficiency on MASTER.
The current state for the segmented_changelog crate in Mononoke is that it does
not assign NON_MASTER commits. It doesn't need to right now. We want to
establish a baseline with the MASTER group. It was however possible for the
on demand update dag to assign commits that were no in the master group to the
master group because no explicit checks were performed. That could lead to
surprising behavior.
At a high level, the update logic that we want is: 1. assign the master
bookmark changeset to the MASTER group, 2. assign other commits that we need to
operate on to the NON_MASTER group. For now we need 1, we will implement 2
later.
Reviewed By: krallin
Differential Revision: D27070083
fbshipit-source-id: 922bcde3641ca25512000cd1a912c5b399bdff4b
Summary:
Pull in SegmentedChangelogConfig and build a SegmentedChangelog instance.
This ties the config with the object that we build on the servers.
Separating the instatiation of the sql connections from building any kind of
segmented changelog structure. The primary reason is that there may be multiple
objects that get instantiated and for that it is useful to be able to pass
this object around.
Reviewed By: krallin
Differential Revision: D26708175
fbshipit-source-id: 90bc22eb9046703556381399442117d13b832392
Summary:
This was lost somehow. I probably incorrectly resolved some conflict when
rebasing a previous change.
Reviewed By: quark-zju
Differential Revision: D27146022
fbshipit-source-id: 13bb0bb3df565689532b2ab5299cd757f278f26e
Summary:
the reclone option code has landed for fbclone, so now we can direct
users there first, so they don't have to go through all these steps
(won't land until I check that this option has actually made it to production)
I also updated the wiki this points to tell users to use `eden list` to detect
EdenFs checkouts instead of looking for .eden, as these steps are also for when
an EdenFS checkout is borked and needs a reclone and `eden list` more reliably
works in this situation.
Reviewed By: StanislavGlebik
Differential Revision: D26435380
fbshipit-source-id: 9153e730e1be949d130af85d604623d2bfbd3990
Summary:
Some of our subprocess calls are running into dylib errors. The cause looks to
be related to our environment variables. We already have environment hygenics
for buck, so lets borrow this to use elsewhere.
This is to fix prefetch profile fetching on mac, but I ran into another error
when testing `eden du --clean`.
Reviewed By: genevievehelsel
Differential Revision: D27135268
fbshipit-source-id: 3955ddefc5e9ff60e966f63f7dc65ef737186464
Summary:
This creates .hg directories when there is no repository at all, which
breaks jf submit in git repos. D26801059 is the real fix, but it has some other
complications. Let's drop the creation here, since it really isn't necessary.
Reviewed By: quark-zju
Differential Revision: D27134087
fbshipit-source-id: d15048b2d1022d38393b62cc02ebf022e617ed4f
Summary:
In D26945466 (7a3539b9c6) I started to use correct repo name for backup repos whenever we
sync an entry. However most of the time sync job is idle, and while doing so it
also logs a heartbeat to scuba table. But it was using wrong repo_id for that
(i.e. for instagram-server_backup it was using instagram-server repo_id). This
diff fixes that.
Reviewed By: krallin
Differential Revision: D27123193
fbshipit-source-id: 80425a56ad0a432180f420f5c7957105407e0fc9
Summary:
Previously the code would result an exception raised while handling another
exception which is a little confusing. This diff fixes that.
Reviewed By: chadaustin
Differential Revision: D27100659
fbshipit-source-id: 8c6be4df62214c8e8d778478de66f271f7b84d3c
Summary:
When using `log -r REVS` with filtering flags like `-u`, `-d`, preserve the
prefetch information by using the `revs(subset=subset)` API.
Reviewed By: sfilipco
Differential Revision: D27119174
fbshipit-source-id: 8483d7113cfc819c6053d1429221588c3a917c12
Summary:
This allows specifying subsets. So we can rewrite:
revs & repo.revs(expr)
to:
repo.revs(expr, subset=revs)
The latter will apply prefetch tweaks on subset when evaluating the revset
expr, while the former will lose prefetch information.
There might be a way to make `revs & repo.revs(expr)` do the right thing
for pre-fetching, too. But that could be more complicated w.r.t `&` fast
paths, over fetching (?), etc. For now I just took the fix that looks more
obvious to me.
Reviewed By: sfilipco
Differential Revision: D27119175
fbshipit-source-id: 2629d21594cf97d7c0f63cf085a2c427d8782e58
Summary:
The filteredset can often be expensive filtering commits like `hg log -u foo`,
`hg log -d '2010-1-1'`. Add a progress bar to show what's going on.
Reviewed By: sfilipco
Differential Revision: D27119176
fbshipit-source-id: 458fbf331978b26e78e6a85fb194ae8b12b949d6
Summary:
`CommitHashToLocationRequestBatch` and `CommitRevlogDataRequest` implemented `FromJson` but not `ToJson`. The latter is required so that the EdenAPI client can log JSON representations of all request types.
In the process of adding tests for those two types, I discovered a bug in `CommitLocationToHashRequestBatch`'s `ToJson` implementation which prevented it from passing JSON roundtrip tests (which I added because this type didn't have test coverage here). The bug is now fixed and the roundtrip test passes.
Reviewed By: krallin
Differential Revision: D27105414
fbshipit-source-id: d2e23e21f996ac8062bca948ed9abe3779e4669d
Summary: Use a macro to define quickcheck tests that perform roundtrip conversions for EdenAPI types that implement `ToJson` and `FromJson`. This makes it easier to add additional roundtrip tests for new types.
Reviewed By: krallin
Differential Revision: D27105416
fbshipit-source-id: c6059f9527d57b88cd6e3d1647fa89d8f826b1e3
