Summary:
Add an `edenapi.logdir` config option, which, when set, will cause the EdenAPI client to write a JSON version of every request it sends to the specified directory. This is intended for use when reproducing user issues (so that the developer can inspect and replay the requests that were sent).
The JSON request files can be directly given to the `edenapi_cli` or converted to CBOR using the `make_req` tool for manual testing with `curl`.
The approach right now is a bit simplistic, in that we just write a JSON file whenever the EdenAPI client makes a request. I'm open to suggestions on how to improve this. (For example, if there were a way for us to always have a record of the last N requests, that would be helpful for debugging user issues.)
Reviewed By: DurhamG
Differential Revision: D27145093
fbshipit-source-id: 3834c2052b0c5efa05d1d209962953b29f545a3f
Summary: noticed these in passing while working on previous diff
Reviewed By: mitrandir77
Differential Revision: D27227682
fbshipit-source-id: e7858c81951b780722b0836ecf6ee72aeb1ffa09
Summary:
The latest git prints extra hints:
$ git init repo
+ hint: Using 'master' as the name for the initial branch. This default branch name
+ hint: is subject to change. To configure the initial branch name to use in all
+ hint: of your new repositories, which will suppress this warning, call:
+ hint:
+ hint: git config --global init.defaultBranch <name>
+ hint:
+ hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
+ hint: 'development'. The just-created branch can be renamed via this command:
+ hint:
+ hint: git branch -m <name>
Initialized empty Git repository in $TESTTMP/repo/.git/
Silent it to make the test pass.
Reviewed By: akushner
Differential Revision: D27235547
fbshipit-source-id: 6f7c8da3ac1e01ee9f57730e1586fa2053f9ca98
Summary:
We need to bump SCS counters expressing Mononoke's QPS. They will look something like:
`requests:mononoke:oregon:carolina` for requests coming from proxygen in prn and mononoke in frc.
CSLB expects regions' full names.
We're getting src region from proxygen as a header.
Reviewed By: krallin
Differential Revision: D27082868
fbshipit-source-id: 12accb8a9df5cf6a80c2c281d2f61ac1e68176d1
Summary:
Add `repo_derived_data`. This is a new struct that encapsulates derived data
configuration and lease operations, and will be used in the facet-based
construction of repositories.
Reviewed By: ahornby
Differential Revision: D27169431
fbshipit-source-id: dee7c032deb93db8934736c111ba7238a6aaf935
Summary:
Add `repo_identity`. This is a new struct that encapsulates repository
identity and will be used in the facet-based construction of repositories.
Reviewed By: ahornby
Differential Revision: D27169445
fbshipit-source-id: 02a435bba54a633190c6d2e4316e86726aecfdf0
Summary:
In preparation for making `BlobRepo` buildable by facet factories, restore
`BlobRepo` members that had been converted to `TypeMap` attributes back into
real members.
This re-introduces some dependencies that were previously removed, but this
will be cleaned up when crates no longer have to depend on BlobRepo directly,
just the traits they are interested in.
Reviewed By: ahornby
Differential Revision: D27169422
fbshipit-source-id: 14354e6d984dfdd2be5c169f527e5f998f00db1e
Summary:
Resolve a circular dependency whereby `BlobRepo` needs to depend on
`Arc<dyn SegmentedChangelog>`, but the segmented changelog implementation
depends on `BlobRepo`, by moving the trait definition to its own crate.
Reviewed By: sfilipco
Differential Revision: D27169423
fbshipit-source-id: 5bf7c632607dc8baba40d7a9d65e96e265d58496
Summary:
a workspace could be renamed from a different machine
educate users that they need to switch to a valid workspace
Reviewed By: markbt
Differential Revision: D27155544
fbshipit-source-id: eed066f2f3e6ebf99732499fdb355f8aebb4c1df
Summary:
this could happen if it has been removed from a different machine
In this case, we should educate the users about the '--force' option
Reviewed By: markbt
Differential Revision: D27155423
fbshipit-source-id: 41cc3ac769dfd4145031fef687e8069d0ef8f4c9
Summary:
provide better ux for switching workspace if the current has been renamed from another machine
sometimes users rename a workspace but it's unclear for them how to switch other machines to the new one
we should educate them about '--force' option
Reviewed By: markbt
Differential Revision: D27117194
fbshipit-source-id: faef1cf9ce64f054f715ef3683a133d3088ddc72
Summary:
We've been seeing flaky test output due to python2 and python3 stdout being different
Bring getdeps builds unambiguously onto python3.
Reviewed By: mitrandir77
Differential Revision: D27190200
fbshipit-source-id: e53327fc3293a0ccdf88d8a199b66eddb4b9b8e4
Summary: There were a few remaining python2 parts to remove and the error case from hgcommand.run() had wrong encoding
Reviewed By: mitrandir77
Differential Revision: D27194046
fbshipit-source-id: 8c154204991e5a587f70ae4b18c73c4f005880ab
Summary:
Before I use a new command line argument I need to add it first.
This one will be set to the top level smc tier which is mononoke, mononoke.experimental, etc.
It will be used to bump SCS counters consumed by CSLB.
I don't want this to be optional because I don't want people to accidentally turn off publishing SCS counters representing QPS. On the other hand I don't want people who run it locally to pollute prod counters (in the script that sets this up locally its set to some dommy tier).
Reviewed By: krallin
Differential Revision: D27190020
fbshipit-source-id: 472011f0584012b036efdd015565cda3b3c029e9
Summary: For the prefetch-profile feature, added a matching enable option to go with the existing disable option. Enable temporarily enables the prefetch-profile feature for cases where a user has already used disable.
Reviewed By: kmancini
Differential Revision: D27191711
fbshipit-source-id: cf9cb7d4054e9fcee66bdb5f205562ec79c2757e
Summary:
When copying files from Finder to EdenFS, Finder appears to be issuing a couple
of getxattr calls to EdenFS, however having these being denied by osxfuse
causes the copy to fail, resulting in a subpar experience.
Reviewed By: fanzeyi
Differential Revision: D27137328
fbshipit-source-id: 66bcc2e316d832385da05f0768da9f82ab36a374
Summary:
This is unused, and the Thrift file mentions that it should be fine to remove
in July 2020. It's now March 2021, time to kill it.
Reviewed By: chadaustin
Differential Revision: D26852134
fbshipit-source-id: 2872185edd834f889b78802210071d16b881e14c
Summary: It's best to test for a regular file by comparing dtype_t, not mode_t directly.
Reviewed By: chadaustin
Differential Revision: D26851735
fbshipit-source-id: 62d3833a53748ea9b45794ec6390d158e888c241
Summary: It doesn't have to be optional
Reviewed By: johansglock
Differential Revision: D27188336
fbshipit-source-id: 796778484aa4c1a455a356b0f412e2abe5522ed0
Summary:
Like it says in the title. See the previous diff for context. This is a little
more complex than I'd like, and IMO largely had to do with the fact that we
have one method for mapping Bonsais AND hg, and that's a bit messy. I think
this is worth refactoring, but in the interest of keeping the diffs small, this
right now just ports the current API to use rendezvous under the hood.
Reviewed By: StanislavGlebik
Differential Revision: D27010315
fbshipit-source-id: 9bef893dd37db244b0b91f5de4fa79d0fdc47228
Summary:
This introduces a basic building block for query batching. I called this
rendezvous, since it's about multiple queries meeting up in the same place :)
There are a few (somewhat conflicting) goals this tries to satisfy, so let's go
over them:
1), we'd like to reduce the total number of queries made by batch jobs. For
example, group hg bonsai lookups made by the walker. Those jobs are
characterized by the fact that they have a lot of queries to make, all the
time. Here's an example: https://fburl.com/ods/zuiep7yh.
2), we'd like to reduce the overall number of connections held to MySQL by
our tasks. The main way we achieve this is by reducing the maximum number of
concurrent queries. Indeed, a high total number of queries doesn't necessarily
result in a lot of connections as long as they're not concurrent, because we
can reuse connections. On the other hand, if you dispatch 100 concurrent
queries, that _does_ use 100 connections. This is something that applies to
batch jobs due to their query volume, but also to "interactive" jobs like
Mononoke Server or SCS, just not all the time. Here's an example:
https://fburl.com/ods/o6gp07qp (you can see the query count is overall low, but
sometimes spikes substantially).
2.1) It's also worth noting that concurrent queries are often the result of
many clients wanting the same data, so deduplication is also useful here.
3), we also don't want to impact the latency of interactive jobs when they
need to a little query here or there (i.e. it's largely fine if our jobs all
hold a few connections to MySQL and use them somewhat consistently).
4), we'd like this to make it easier to do batching right. For example, if
you have 100 Bonsais to map to hg, you should be able to just map and call
`future::try_join_all` and have that do the right thing.
5), we don't want "bad" queries to affect other queries negatively. One
example would be the occasional queries we make to Bonsai <-> Hg mapping in
`known` for thousands (if not more) of rows.
6), we want this to be easy to incorporate into the codebase.
So, how do we try to address all of this? Here's how:
- We ... do batching, and we deduplicate requests in a batch. This is the
easier bit and should address #1, #2 and #2.1, #4.
- However, batching is conditional. We notably don't batch very large requests
with the rest (addresses #5). We also don't batch small queries all the time:
we only batch if we are observing a throughput of queries that suggests we
can find some benefit in batching (this targets #3).
- Finally, we have some utilities for common cases like having to group by repo
id (this is `MultiRendezVous`), and this is all configurable via tunables
(and the default is to not do anything).
Reviewed By: StanislavGlebik
Differential Revision: D27010317
fbshipit-source-id: 4a2397255f9785c6722c02e4d419438fd0aafa07
Summary:
Generic structs can't be used as facets, which prevents `RepoBonsaiSvnrevMapping` being
used as a facet. Currently we only use it with `Arc<dyn BonsaiSvnrevMapping>` anyway,
so make that official by removing the generic parameter.
Reviewed By: ahornby
Differential Revision: D27169420
fbshipit-source-id: 908b1555341652e72adad087bc0b77565cd75b9d
Summary:
It's really annoying that we're not throwing away the cause here, because when
you get a crash now you get zero context. This fixes that. I'd rather not say
how long it took me to find this line...
Reviewed By: StanislavGlebik
Differential Revision: D27168808
fbshipit-source-id: ac3312c2f3b328f08929be499c45c4462cb159a3
Summary:
EdenFS doesn't register itself against the portmap client, and on some system
where it is not started, it appears to not work reliably, crashing EdenFS early
at startup. For now, let's only build it when services need to be registered.
Reviewed By: genevievehelsel
Differential Revision: D27162906
fbshipit-source-id: cc2a8a588a756e54253da31f9bc00fbe4e5312d9
Summary:
In a bunch of places, the code assumes that an EdenMount is associated with a
Fuse channel. With NFS, that's no longer the case, thus let's make sure to
check the return value of mount->getFuseChannel(). In the case where it will
make sense to have something for NFS, I've either added an EDEN_BUG, or a TODO,
so we can come back to it later.
Reviewed By: chadaustin
Differential Revision: D26836431
fbshipit-source-id: c061b8f20199e5af3139a5003827f184f6eac8d4
Summary:
In the case where EdenFS would fail to initialize very early, the optional
server would be empty, and thus trying to dereference it would lead to EdenFS
crashing with no good error message. Let's simply test for this.
Reviewed By: genevievehelsel
Differential Revision: D27162907
fbshipit-source-id: a078b9995a94e5a86cf2893cbebfe7f5ca6d064e
Summary:
Currently we can only limit which users are allowed to move a bookmark by a
regex. We also want to allow specifying a hipster group.
Reviewed By: krallin
Differential Revision: D27156690
fbshipit-source-id: 99a5678a82f4c34ed2e57625361ba7cdb08ed839
Summary:
See the previous diff for context. Disable the Rust progress for external
pager.
Reviewed By: kulshrax
Differential Revision: D27149241
fbshipit-source-id: 4260a8be55bbfa648d8910f021195e9d11bdab73
Summary:
When testing "disable_progress" with chg (next diff) I found it was not
effective because there are 2 separate IO structs. The one we disable
from Python is different from the one the Rust progress thread uses.
I traced it down here. Since the Python IOs are just wrappers of
Rust IOs in the chg use-case. There is no need to recreate an IO
struct.
The "creating IO" struct is still useful, for things like "-t.py" testing where
the output needs to be captured into different Python variables per different
commands.
Reviewed By: DurhamG
Differential Revision: D27149243
fbshipit-source-id: 6e27adcc9f48b21fc24fba120be8c4a8fef1f909
