Summary:
It's nice to have these functions to open source and target repos.
Previously we always had to get repo id first, and then call
open_repo_internal_with_repo_id
Reviewed By: yancouto
Differential Revision: D30866314
fbshipit-source-id: dd74822da755de232f4701f8523088e0bb612cb9
Summary:
D30829928 (fd03bff2e2) updated some of Mononoke's integration tests to take into
account whitespace changes. However, it removed the globs from some parts of
the tests.
As the assigned port changes on each test run, the globs are required. Add them
back in again, as well as fix up some whitespace in a test.
Reviewed By: markbt
Differential Revision: D30866884
fbshipit-source-id: 1557eee2143a2459a6412b8649e7e3dce5a607a4
Summary:
It's nice to have that can quickly count and print stats about a commit. I'm
using it now to understand performance of derived data.
Reviewed By: ahornby
Differential Revision: D30865267
fbshipit-source-id: 26b91c3c05a1c417015b5be228796589348bf064
Summary:
`rust_include_srcs` is supported on `thrift_library` as a way of including other Rust code in the generated crate, generally used to implement other traits on the generated types.
Adding support for this in autocargo by copying these files into the output dir and making sure their option is specified to the thrift compiler
Reviewed By: ahornby
Differential Revision: D30789835
fbshipit-source-id: 325cb59fdf85324dccfff20a559802c11816769f
Summary:
The default Windows encoding can't handle some unicode characters
apparently, so let's use utf-8 by default.
Reviewed By: quark-zju
Differential Revision: D30850982
fbshipit-source-id: 51a7fdf5464d075549afe4f0bcd307c0f2eb7fa0
Summary:
Add impls for Layer for Box/Arc<L: Layer> and <dyn Layer>. Also a pile of other
updates in git which haven't been published to crates.io yet, including proper
level filtering of trace events being fed into log.
Reviewed By: dtolnay
Differential Revision: D30829927
fbshipit-source-id: c01c9369222df2af663e8f8bf59ea78ee12f7866
Summary:
Bump all the versions on crates.io to highest to make migration to github
versions in next diff work.
Reviewed By: dtolnay
Differential Revision: D30829928
fbshipit-source-id: 09567c26f275b3b1806bf8fd05417e91f04ba2ef
Summary:
We don't need to pass the bubble id to the server, it can find it from the changeset id.
This fixes a TODO I added previously, and should make the `restore` command complete.
Reviewed By: ahornby
Differential Revision: D30609423
fbshipit-source-id: d1c8eb0e0556069fa408520a0aea91a0f865fbe1
Summary:
Uses the endpoint added on previous diffs to download the snapshot files to the repo, and adds them correctly to the snapshot restore.
This should almost complete the `snapshot restore` command, missing is getting the bubble id from the snapshot hash.
{gif:sqc6yx6c}
Reviewed By: StanislavGlebik
Differential Revision: D30583038
fbshipit-source-id: 6549a52f767c50444c316b358d9704bc4a136934
Summary:
This adds the `downloadfiles` method to the python EdenApi wrapper.
It uses multiple calls to the endpoint added on previous diffs to download each file and place it somewhere on the repo. It also does deduplication of downloads.
Reviewed By: StanislavGlebik
Differential Revision: D30582638
fbshipit-source-id: 34e864d03c0e48a7605ee8e4e92376881dbb2de9
Summary:
When using hashbinary with a removed/moved file, hg throws with `TypeError: object supporting the buffer API required` this is because we are trying to `sha1(None)`.
This diff falls back to the `Binary file %s has changed` message when we have a removed file.
Reviewed By: quark-zju
Differential Revision: D30845897
fbshipit-source-id: a3d2b7d11d9c1ca3855140c9abd7550cf7076bbc
Summary: This adds the support for FS events logging for NFS. For context, each type of event is assigned a sampling group that determines its sampling rate. In TraceBus subscription callback, events are sent to `FsEventLogger` to be sampled and logged through `HiveLogger`.
Reviewed By: xavierd
Differential Revision: D30843863
fbshipit-source-id: 65394d31b1197efd69c7fd4c1b24562f5abd5785
Summary:
Previously, it was only possible to register event listeners for request completion on the `HttpClient` itself, rather than on individual `Request`s. This diff adds similar event listeners to `Request`s themselves, so that its possible to register a callback to fire when any request completes, regardless of whether it was sent via an `HttpClient` or as a one-off.
This is similar to `RequestCreationEventListeners`, which run for the creation of every request, whether or not the request is associated with a client.
Notably, to avoid circular references the new event listeners take a `RequestInfo` argument instead of a `RequestContext` (since the listeners are themselves stored inside the `RequestContext`). In practice, the `RequestInfo` should contain all of the information one might want to access about the request.
Reviewed By: quark-zju
Differential Revision: D30831840
fbshipit-source-id: 77ca9dc5fd9f8fc5ee60319baabd77171af70d45
Summary:
The content store repair binding ate the entire repair message, making
it hard to debug when it wasn't working.
Reviewed By: quark-zju
Differential Revision: D30824740
fbshipit-source-id: 52dbfe79f2dd1568285cda63fb54cacf532aa4a1
Summary:
Make `verify` check the lazy changelog properties:
- Universal id <-> name mappings are known locally.
- Segments are sane (ex. high-level segments are built from low-level and there
are no cycles)
With `--dag`, also check the graph with a newly cloned remote graph.
This just calls the verification logic added in Rust `dag` crate to do the
heavy lifting.
Differential Revision: D30820773
fbshipit-source-id: 8f62f41738c3c8e3fe88442860a83fdb4944f178
Summary:
In certain situations, users may cause EdenFS to falsely return a path not exist result while the path is available. Windows will cache that and causing subsequent access to that file to automatically return a file not exist error.
We currently only invalidate this negative cache during checkout and rebooting the machine as the cache is even kept during EdenFS restarts. In this diff, we starts to invalidate the negative path cache at startup so if the user ever had issues an `eden restart` would be sufficient to fix.
Reviewed By: xavierd
Differential Revision: D30814059
fbshipit-source-id: 53283f471702762b2eed0c5d0f6a9cc49f4db739
Summary:
This adds the plumbing to access download a file using the endpoint from the previous diff via the EdenApi trait, which does the actual http request.
It concats the stream into a Bytes object and returns it.
Reviewed By: StanislavGlebik
Differential Revision: D30582422
fbshipit-source-id: ed0fe5e34e3fecc6c1b26d2dceb322dfcf5f8e37
Summary:
This diff adds an endpoint `/download/file` that allows to download a file given an upload token.
This will be used for snapshots, as we need to download the snapshot changes, and there's no way to do that right now.
Other options, and why I didn't do them:
- Using the existing `/files` endpoint: Not possible, as it needs hg filenodes and we don't have those.
- Returning the file contents in the fetch_snapshot request: Might make the response too big
- Returning just a single Bytes instead of a stream: I thought streaming would be preferred, and more future proof. In the stack I still put everything in memory in the client, but maybe in the future it should be possible to stream it directly to the file. I'm happy to remove if preferred, though.
Reviewed By: StanislavGlebik
Differential Revision: D30582411
fbshipit-source-id: f9423bc42867402d380e831bc45d3ce3b3825434
Summary: This proved useful couple of times when folks experienced problems with the agent.
Reviewed By: ahornby
Differential Revision: D30837676
fbshipit-source-id: aec769f60a09ecb83857e6e60d49a5662b4ce0b2
Summary:
Add back the octopus merge support for revlog.
This recommits D30686451 (b13579fdf9) and D30686450 (7eb11cb392) as-is, with updates to test files.
Original commit changeset: 9f213766e7c4
Reviewed By: StanislavGlebik
Differential Revision: D30784681
fbshipit-source-id: ace0c317652ad8b657c8edd9a0130532dad53078
Summary:
As far as I could tell, this was legacy from some refactorings.
It was only used in one place, and it was easy to fix.
Also, if we really need it in the future, we can probably use `#[auto_impl]` instead of doing it manually.
Reviewed By: StanislavGlebik
Differential Revision: D30574803
fbshipit-source-id: 20715364713775818fe0e83844637f48b310d87f
Summary: createremote only worked from root of the repo. This fixes it, and tests that in the integration test
Reviewed By: StanislavGlebik
Differential Revision: D30546582
fbshipit-source-id: 84aa304d346e448b44e5d7fb9e9607d84a67da25
Summary:
This adds basic logic for `snapshot restore` command.
- It updates to the parent of the snapshot
- It loads the snapshot changes
For now I did not do changes/tracked changes, as it will need to download the file contents, which will need a new edenapi endpoint, so I'll leave it for a future diff. It just restores your deleted files for now.
Reviewed By: StanislavGlebik
Differential Revision: D30543507
fbshipit-source-id: 080588ceff0ecd595ce739044f0d4118fb8e1a3f
Summary:
log sync reason for `hg cloud sync`.
This will help us to investigate issues better and measure impact for new Eden Api Uploads case by case (after amend, rebase, etc) on different platforms.
Reviewed By: yancouto
Differential Revision: D30775519
fbshipit-source-id: 696e954ec8db19226fb67ad0952e23e2b67e9931
Summary:
Put code using the usage service behind an `EDEN_HAVE_USAGE_SERVICE` macro.
Previously the C++ code was simply guarded by a `__linux__` check, and the
CMake code did not have a guard at all. This caused builds from the GitHub
repository to fail on Linux, since the code attempted to use the usage service
client which was not available.
Reviewed By: xavierd
Differential Revision: D30797846
fbshipit-source-id: 32a0905d0e1d594c3cfb04a466aea456d0bd6ca1
Summary:
In the v1 sparse config arrangement, if all rules were excludes then we
would include a default "**" rule. This was always a little confusing and caused
some weird behavior. Let's remove it from the v2 world.
This actually bit us because the fbsource_exclude profile only has excludes,
which caused it to insert a ** include, which pulled in all of fbsource. We
could fix it to only check if a profile is excludes-only once all the transitive
profiles have been loaded, but I think the cleaner fix is to remove this logic
since it's confusing and never actually used in production.
Differential Revision: D30824082
fbshipit-source-id: adcf4c820cc9f7636f79759d03fc0b387b9f55fa
Summary:
Any error inside the decode stream was being propagated up as a decoder
error. This caused higher level code to not handle certain errors appropriately.
For instance, the lfs retry logic only retries for certain classes of curl
errors. So let's propagate up HttpClientErrors as is.
Reviewed By: kulshrax
Differential Revision: D30798108
fbshipit-source-id: 7316f6cdc47de090c202ff6a1f28d0fba60f7a15
Summary:
The previous version had two issues:
1. It's UB to cast uninit away as it may be actually uninitialized.
2. Because of the cast, the buffer was not actually written nor advanced after written to, causing the caller to think nothing was read.
https://docs.rs/tokio/1.11.0/tokio/io/struct.ReadBuf.html
Reviewed By: dtolnay
Differential Revision: D30823808
fbshipit-source-id: d5f67e4c03f1d63f2241421dd35082ee96b5afd8
Summary: For some reason it got broken, need to call `as_ref()` to properly cast type
Reviewed By: quark-zju
Differential Revision: D30740629
fbshipit-source-id: f49275caae9d360859e97c03709a720dabc22e9e
Summary:
LocalStore no longer special-cases Tree objects with kZeroHash
ids. Instead, unconditionally write into LocalStore with the Tree's
hash.
Reviewed By: xavierd
Differential Revision: D29155470
fbshipit-source-id: aee3840fe8dfd7aa46305b6db6f7950efb2e41d2
Summary:
In preparation for expanding to variable-width hashes, rename the
existing hash type to Hash20.
Reviewed By: genevievehelsel
Differential Revision: D28967365
fbshipit-source-id: 8ca8c39bf03bd97475628545c74cebf0deb8e62f
Summary:
Do not assume `changelog.parents` returns 2 items.
This changes the behavior for root commits. `parents()` used to return
`[repo[nullid]]`, now it returns `[]`.
Reviewed By: andll
Differential Revision: D30784684
fbshipit-source-id: 73f58c85457391fb74b96b88dc4dcb69a25e81ac
Summary:
In a future change, `ctx.parents()` returns `[]` instead of `[repo[nullid]]`
for root commits. Make the change to preserve absorb behavior.
Differential Revision: D30816385
fbshipit-source-id: afded91a6e72d4eb54faf87dcdfc52a81ea1d66f
Summary:
In a future change, `ctx.parents()` returns `[]` instead of `[repo[nullid]]`
for root commits. Make the change to preserve rebase behavior.
Differential Revision: D30816386
fbshipit-source-id: ca7c489991ae149c9640b7da0e6e54f76afbc250
Summary:
We're going to change parents() to return an empty list instead of `[nullctx]`
for roots. This change makes it more compatible with upcoming changes.
Reviewed By: andll
Differential Revision: D30787305
fbshipit-source-id: 1de523964faa64a6496a7bb0197af597e393d859
Summary: This will be used by the next change.
Reviewed By: andll
Differential Revision: D30784683
fbshipit-source-id: 59a37c5f428eaf5950584d8f17471d358bfefee7
Summary: Integrate http hash prefix lookup into the pull operation. One unfortunate change here is that if the prefix is ambiguous, we're only able to output possible full hashes as suggestions. Previously we'd also print commit log information. To retain that we'd need to add an error option to the response and have the server send back an error message with the log information or send another request to download the extra information.
Reviewed By: andll
Differential Revision: D30716050
fbshipit-source-id: 33f8bc38b0bfe7fce4ec11cd8def7feda3b3d3da
Summary:
As title, sampling group determines the sampling rate at which an FS event is logged. The higher the sampling group the more heavily its events are dropped, thus, more frequent events are assigned to the higher sampling groups.
I ran activity recorders on a few workflows, buck build, getdepts, and vscode editing and came up with the following assignment. Note that only a subset of events are assigned to a sampling group (so events not included will not be logged) as we just start to tune the sampling rates and these events should be good for a start.
```
Group1 (1/10)
FUSE_MKDIR
FUSE_RMDIR
FUSE_CREATE
FUSE_RENAME
Group2 (1/100)
FUSE_WRITE
FUSE_LISTXATTR
FUSE_SETATTR
Group3 (1/1000)
FUSE_GETXATTR
FUSE_GETATTR
FUSE_READ
FUSE_READDIR
Group4 (1/10000)
FUSE_LOOKUP
```
For reference, here are the counts of FS events of a cold buck build. The frequencies of other workflows are similar.
```
FUSE_LOOKUP 60.09 98733
FUSE_READ 12.80 21037
FUSE_GETXATTR 8.91 14645
FUSE_FORGET 8.01 13162
FUSE_GETATTR 5.55 9116
FUSE_READDIR 3.21 5270
FUSE_LISTXATTR 0.59 969
FUSE_READLINK 0.54 892
FUSE_STATFS 0.21 338
FUSE_WRITE 0.04 64
FUSE_CREATE 0.02 28
FUSE_RENAME 0.01 23
FUSE_SETATTR 0.01 13
FUSE_UNLINK 0.00 6
FUSE_RMDIR 0.00 1
FUSE_MKDIR 0.00 1
FUSE_MKNOD 0.00 1
```
Reviewed By: xavierd
Differential Revision: D30770533
fbshipit-source-id: 90be881ddbeba2113bbb190bdb1e300a68f500a0
Summary: The new `EdenApiHandler` framework for defining EdenAPI endpoints provides a common place where responses are encoded. This diff adds automatic content compression at this point, using the received `Accept-Encoding` header from the request to determine what compression, if any, should be used. As a result, all endpoints that implement `EdenApiHandler` will get compression for free.
Reviewed By: yancouto
Differential Revision: D30553242
fbshipit-source-id: 9eda54cbf81dd1e03abec769744c96b16fad64ea
Summary:
It can sometimes be difficult to work out from the logging which commit cloud
requests came from which client repo. Previously you could often infer it from
the client identities, however if the request is proxied, the originating hostname can be
lost, and it still doesn't handle the case where the host contains multiple
repos.
This diff adds a new `ClientInfo` struct, which is included by the client
on every `get_references` and `update_references` request. This is logged
by the service, allowing us to correlate which client it came from, and what
workspace version the client had at that time.
Reviewed By: StanislavGlebik
Differential Revision: D30697839
fbshipit-source-id: 8fe2e03f0be2f443f8ae1814f083c04ba5d1805e
Summary: It was not used and hard to implement retry with
Reviewed By: yancouto
Differential Revision: D30716647
fbshipit-source-id: a90b629f7758486c9e526d1eaf3fd29da305f2e7
Summary:
D30704344 (5704ad51f6) upgraded tokio for the buck build. We need to do the same for
the non-buck build. This unbreaks hgbuild.
Also clean up some compiler warnings while I'm here.
Reviewed By: fanzeyi
Differential Revision: D30798315
fbshipit-source-id: 47005c7674d87196aab42b3ddf2194acced3bb6c
Summary:
We have two mode of deriving data: the "normal" way and using backfilling.
Backfilling is different from "normal" mode in that it derives a few commits at
once, and saves them all to blobstore at once.
Backfilling mode seemed to have helped us when we need to derive a lot of data
(e.g. backfill the whole repo). But
a) We don't know how much it helps, and we don't know if it depends on the repo
b) We don't know if it helps when we derive data for newly landed commits (i.e.
we use "backfill" mode in derived data tailer to derive data for latest public
commits)
So this diff adds a bit of logging to a separate scuba table so that we can get
an idea about things like:
1) How long does it take to derive a stack of commits?
2) Where do we spend most of the time (e.g. deriving, saving the blobs, saving
the mapping).
Reviewed By: mitrandir77
Differential Revision: D30805504
fbshipit-source-id: d82c905cafa87459990d74769a0dddcc91fac174
Summary:
It allows us to do 3 things:
1) Remove derive function
2) Add support for backfill mode so that we can compare perf with and without
it
3) Use all derived data types, and not just 3 of them
Reviewed By: krallin
Differential Revision: D30804258
fbshipit-source-id: 604723a3d845a60cfd94b4e090a121f5b5191536
Summary: This command can be useful to split a large bonsai commit into a smaller one
Reviewed By: mitrandir77
Differential Revision: D30776789
fbshipit-source-id: dc56d7c51eb0e9e0988dcba868c6008ebf488927
Summary:
While we don't really need it, creation mononoke matches fail if they are not
present. Let's just enable it here - it's not a bad thing to initialize them
Reviewed By: mitrandir77
Differential Revision: D30780463
fbshipit-source-id: c4199c6711ae7bd9641e9f51643b94d020051dbd
Summary: The code for accessing config fields had a lot of repetitive boilerplate. Let's move that to a helper function.
Reviewed By: andll
Differential Revision: D30785932
fbshipit-source-id: fb4d47337a27bd6e75eeb38d5a9d1de5b1fac6ce
Summary: Implement serverside graph endpoint for fetching the mapping of commits to commit parents for the missing segment of a commit graph. This implementation uses the find_commits_to_send method from the get_bundle_response library. What may be missing from pull and the old bundle protocol now is mutation markers.
Reviewed By: yancouto
Differential Revision: D30485672
fbshipit-source-id: ba3a30d9e482d60831cbe7a8e89f20dab947d9a1
Summary:
Since the find_commits_to_send method was added, common is already a
hashset not a vector, so it doesn't needed to be converted to a hashset.
Reviewed By: quark-zju
Differential Revision: D30622028
fbshipit-source-id: e5d1b6c60115d13c906b25142043652ba9e89d70
Summary:
Not flushing the data to disk makes studying performance almost impossible due
to not being able to avoid fetching from the network. By forcing a flush to
disk, we can ensure that data will always be on disk, making performance
measurement easier. This will also prevent users from re-fetching the same data
multiple times.
Reviewed By: fanzeyi
Differential Revision: D30784399
fbshipit-source-id: 0250c209b5f49f95cf2f43873573cacc661a4989
Summary:
Since this method wasn't overriden, EdenFS would never periodically flush data
to disk.
Reviewed By: fanzeyi
Differential Revision: D30784400
fbshipit-source-id: d88e535250a476582868dd82e57137a0ac38f921
Summary: Previously, there were special variants for missing and invalid URLs (since the server URL is presently the only required config option). In order to support other required config options, let's simplify the enum to just have variants for missing and invalid config fields respectively.
Reviewed By: yancouto
Differential Revision: D30745971
fbshipit-source-id: e414ec2fadc5d04e9c788bf290a70f6cf52dbe58
Summary: Saw this when reading related code.
Reviewed By: kulshrax
Differential Revision: D30783665
fbshipit-source-id: f9b598b9301619346972bd0abf893f089d902022
Summary:
I'm not sure what this was for, but it doesn't seem necessary, and removing it simplifies the code a lot, enabling us to make other improvements later.
This is an alternate, less ambitious version of https://www.internalfb.com/diff/D30620443.
Reviewed By: DurhamG
Differential Revision: D30674016
fbshipit-source-id: 17dee50b82c78d31e45492dc23826d8c3fe838e5
Summary:
This test relies on Mononoke, so it fails for make local build/test,
which breaks hgbuild. Let's only enable it for buck tests.
Reviewed By: quark-zju
Differential Revision: D30782799
fbshipit-source-id: 4b543beeb248715702b9072d84cdb8211dcd4a9b
Summary:
Currently there are two things preventing us from running add_sync_target
on existing target:
* already existing bookmark
* already existing config
Both need to be deleted to create new target. This diff removes the second
one to simplify code and make it easier to recreate the target (it's easy to
forget about manually removing the config as they otherwise don't need
human interventions).
Reviewed By: StanislavGlebik
Differential Revision: D30767613
fbshipit-source-id: f951c0e1ef9bde69d805dc911331fcdb220123f2
Summary:
This logic scans all the ancestors of the working copy that are not
ancestor of the graft source and checks their extras. With lazy changelog this
is extremely expensive. Let's just drop this logic.
Reviewed By: quark-zju
Differential Revision: D30734017
fbshipit-source-id: ca5606cea08efe10f29847970379d6bff4eb4aee
Summary:
Update the `Filenodes` trait so that it doesn't require the repository id to be
passed in every method invocation. In practice a filenodes instance can only
be used for a single repo, so it is safer for the implementation to store the
repository id.
At the same time, update the trait to use new futures and async-trait.
Reviewed By: yancouto
Differential Revision: D30729630
fbshipit-source-id: a1f80a299d9b0a99ddb267d1f7093f27cf21f1af
Summary:
Make the derivation process for mercurial changesets and manifests not depend
on `BlobRepo`, but instead use the repo attribute (`RepoBlobstore`) directly.
This will allow us to migrate to using `DerivedDataManager` in preparation
for removing `BlobRepo` from derivation entirely.
Reviewed By: yancouto
Differential Revision: D30729629
fbshipit-source-id: cf478ffb97a919c78c7e6e574580218539eb0fdf
Summary:
Make the derivation process for blame and deleted files manifest not depend
on `BlobRepo`, but instead use the repo attribute (`RepoBlobstore`) directly.
This will allow us to migrate to using `DerivedDataManager` in preparation
for removing `BlobRepo` from derivation entirely.
A `BlobRepo` reference is still needed at the moment for derivation of the
unodes that these depend on. That will be removed when `DerivedDataManager`
takes over co-ordination of derivation.
Reviewed By: yancouto
Differential Revision: D30729628
fbshipit-source-id: 4504abbe63c9bf036d69cb4341c75b13061fae18
Summary:
Make the derivation process for fsnodes and skeleton manifests not depend on
`BlobRepo`, but instead take the `DerivedDataManager` from the `BlobRepo` and
use that instead. This is in preparation for removing `BlobRepo` from
derivation entirely.
Reviewed By: yancouto
Differential Revision: D30301855
fbshipit-source-id: a2ed1639526aad9ddbe8429988043f0499f7629c
Summary:
Make the derivation process for unodes not depend on `BlobRepo`, but instead
take the `DerivedDataManager` from the `BlobRepo` and use that instead. This
is in preparation for removing `BlobRepo` from derivation entirely.
Reviewed By: yancouto
Differential Revision: D30300408
fbshipit-source-id: c35e9e21366de74338f453aaf6be476e7305556d
Summary:
The derived data manager also has a reference to the repo blobstore. This must
also be overridden when we override the blobstore.
Reviewed By: yancouto
Differential Revision: D30738354
fbshipit-source-id: b0e16ef810c5244cd056a3c9e5b9ceaaddb5ecea
Summary:
modernise tests (removing disabling treemanifest)
treemanifest is the default now, so it shouldn't be disabled unless absolutely necessary.
Also, if we would like to switch some of the tests to Mononoke, it shouldn't be disabled.
Only two tests left with it in commitcloud. Those two a bit harder to fix.
```
devvm1006.cln0 {emoji:1f440} ~/fbsource/fbcode/eden/scm/tests
[139] → rg 'disable treemanifest' | grep cloud
test-commitcloud-backup-bundlestore-short-hash.t: $ disable treemanifest
test-commitcloud-backup.t: $ disable treemanifest
```
Reviewed By: kulshrax
Differential Revision: D30278754
fbshipit-source-id: cf450084669c2b6b361cd34952bf986e913de1a8
Summary:
I want to use `ReplicaFirst` read connection type since `ReplicaOnly` is a bit too restrictive.
We've had 2 MySQL SEVs this year when all the replicas went down crashing our services despite the primary instance working normally.
There was also a case when I've deleted too much rows at once and all replicas went down due to replication lag (I know better now)
RFC
- Yay or Nay?
- Should I expand `ReadConnectionType` to mirror all options of `InstanceRequirement`?
- Perhaps it's worth moving it into the `common/rust/shed/sql` crate?
I kept cleaning up all the usages out of this diff to keep the changes minimal for RFC
Differential Revision: D30574326
fbshipit-source-id: 1462b238305d47557372afe7763096c53df55f10
Summary:
Segmented changelog seeder spends a significant chunk of time fetching
changesets. By saving them to file we can make reseeding significantly faster.
Reviewed By: farnz
Differential Revision: D30765374
fbshipit-source-id: 0e6adf12e334ad70486145173ae87c810880988a
Summary:
In backfill_derived_data we had a way to prefetch a lot of commits at once, so
that backfill_derived_data doesn't have to do it on every startup.
I'd like to use the same functionality in segmented changelog seeder, so let's
move it to the separate binary.
Reviewed By: mitrandir77, farnz
Differential Revision: D30765375
fbshipit-source-id: f6930965b270cbaae95c3ac4390b3d367eaab338
Summary: ContentDataStore is meant to be implemented local-only. Fetching remotely seems to cause the issue observed in https://fb.workplace.com/groups/scm/permalink/4192991577417097/ (though I'm not quite sure why yet)
Reviewed By: kmancini
Differential Revision: D30744817
fbshipit-source-id: 68875a4912905f9b8f88cf4be804c5d988c3905d
Summary: If the bind unmount fails in in the privhelper, theres a possibility of infinite recursion in this method. This adds a flag to indicate if we've tried the bind unmount before.
Differential Revision: D30732857
fbshipit-source-id: 6ee887d211977ee94c8e66531287f076a7e61a2c
Summary:
It sounds like macOS has a bug where an APFS subvolume may be falsely created.
Let's retry with the hope that the retry will succeed.
Differential Revision: D30657706
fbshipit-source-id: 60bc74f789a0d34b2be53073103b95474a9a18e6
Summary: This is regenerated rust lib using the latest compiler
Reviewed By: krallin
Differential Revision: D30720130
fbshipit-source-id: 3d3389ec8504568fc356dda1577e1f7801cb7e96
Summary:
~~Also enable the `derive` feature so it isn't necessary to separately
depend on `strum_macros`.~~
This turns out to break a lot.
Reviewed By: dtolnay
Differential Revision: D30709976
fbshipit-source-id: a9181070b8d7a8489eebc9e94fa24f334cd383d5
Summary: Move `edenapi::Client`'s internals to an `Arc<ClientInner>`. This makes the client `Clone`-able while sharing the same underlying state. This is particularly useful for scenarios where `Future`s or `Stream`s returned by the client need to hold a reference to the client itself (e.g., in order to issue subsequent HTTP requests).
Differential Revision: D30729803
fbshipit-source-id: c97e700c9e3702f818eb86ded1a46f920a55cfd1
Summary: The `Fetch<T>` type has basically turned into the canonical type EdenAPI for all EdenAPI responses. Originally, this type was merely an implementation detail (essentially just a named tuple returned by the `fetch()` method, hence the name), but given its prominence in the API, the name is confusing. As we add more functionality and usage to this type, it makes sense to give it a more suitable name.
Differential Revision: D30730573
fbshipit-source-id: 7acd2a86b55bdfc186bd9110f6a99333df9d29d3
Summary:
Some of the method names used internally by `edenapi::Client` are a bit terse.
This was OK back when there were only handful of private methods which were used by a small number of API methods that were doing more or less the same thing (sending concurrent POST requests for a set of keys).
Today, there are way more API methods, most of which set up requests in different ways. As such, it makes sense to give these older private methods more explicit and descriptive names so that their intended usage is clear.
Differential Revision: D30729802
fbshipit-source-id: 5adfd8e7ba153df8c036e4dbb312f95b9b1d7335
Summary: Allow repack to be called on treescmstore via the ContentStore shim like filescmstore is already supported.
Reviewed By: andll
Differential Revision: D30687145
fbshipit-source-id: 7559af08e98cfb22da6dbf45dc1746312b1e6d28
Summary:
Provide a basic implementation of the LegacyStore trait for TreeStore to allow repack calls to be forwarded to the fallback ContentStore for trees.
Repack will be removed entirely before contentstore is deleted, and the `unimplemented` methods are never called, so this should be safe.
Reviewed By: andll
Differential Revision: D30687136
fbshipit-source-id: d238d70fbf6be5c25c2e1c9610430a53d031bf3b
Summary: Looks like it was lost during the last refactoring, let's add it back.
Reviewed By: farnz
Differential Revision: D30728456
fbshipit-source-id: 20c638b3c5a8664f2367f871cd29a793fd897de3
Summary:
Some users have reported errors of the form:
```
error.HttpError: [65] Send failed since rewinding of the data stream failed (seek callback returned error 2)
```
These are caused by the fact that we're passing the HTTP request body directly to libcurl in memory rather than via a file, but we haven't implemented the `seek()` method necessary for libcurl to retransmit the data if needed. This diff implements the method.
Reviewed By: DurhamG
Differential Revision: D30654625
fbshipit-source-id: f21a067ad02ee540b86cf2e6eff2c6f08f45a3e4
Summary:
Like it says in the title, this updates us to use Daemonize 0.5, though from
Github and not Crates.io, because it hasn't been released to the latter yet.
The main motivation here is to pull in
https://github.com/knsd/daemonize/pull/39 to avoid leaking PID files to
children of the daemon.
This required some changes in `hphp/hack/src/facebook/hh_decl` and `xplat/rust/mobium` since the way to
run code after daemonization has changed (and became more flexible).
Reviewed By: ndmitchell
Differential Revision: D30694946
fbshipit-source-id: d99768febe449d7a079feec78ab8826d0e29f1ef
Summary:
At the moment when segmented changelog is updated and/or reseeded mononoke
servers can pick it up only once an hour (this is a current reload schedule)
or when mononoke server is restarted. However during production issues (see
attached task for an example) it would be great to have a way to force all
servers to reload segmented changelog.
This diff makes it possible to do so with a tunable. Once tunable changes its
value then monononoke servers almost immediately (subject to jitter) reload it.
This implementation adds a special loop that polls tunables value and reloads
if it changes. Note that in theory it could avoid polling and watch for configerator
changes instead, but it would be harder to implement and I decided that it's
not worth it.
Reviewed By: farnz
Differential Revision: D30725095
fbshipit-source-id: da90ea06715c4b763d0de61e5899dfda8ffe2067
Summary:
Previously, extremely large prefetch calls could cause an OOM if the requested files were all fetched remotely from EdenApi, in which case the memory would remain in use until the entire batch had been fetched.
With this change, at most 1000 EdenApi files will be held in memory at once (or 10GB of memory). This is a stop-gap solution, a better approach would be to avoid storing all EdenApi files in memory after a certain amount, or allow the batch fetching implementation to understand we're only prefetching, and thus avoid reading anything back from disk or storing EdenApi files in memory unnecessarily.
Reviewed By: andll
Differential Revision: D30686054
fbshipit-source-id: 022e353760c515961a8956f7958b43f429143971
Summary:
Manual component version update
Bump Schedule: https://www.internalfb.com/intern/msdk/bump/?schedule_fbid=342556550408072
Package: https://www.internalfb.com/intern/msdk/package/181247287328949/
Oncall Team: rust_foundation
NOTE: This build is expected to expire at 2022/09/01 09:14AM PDT
---------
New project source changes since last bump based on D30663071 (08e362a355e0a64a503f5073f57f927394696b8c at 2021/08/31 03:47AM -05):
| 2021/08/31 04:41AM -05 | generatedunixname89002005294178 | D30665384 | [MSDK] Update autocargo component on FBS:master |
| 2021/08/31 07:14PM PDT | kavoor | D30681642 | [autocargo] Make cxx-build match version of cxx |
| 2021/09/01 04:05PM BST | krallin | D30698095 | autocargo: include generated comment in OSS manifests |
---------
build-break (bot commits are not reviewed by a human)
Reviewed By: farnz
Differential Revision: D30717040
fbshipit-source-id: 2c1d09f0d51b6ff2e2636496cf22bcf781f22889
Summary: Keep two versions of fbthrift_ext that one with tokio-0.2 and the other with tokio-1.x. This diff is just renaming.
Reviewed By: dtolnay
Differential Revision: D30558441
fbshipit-source-id: bfe7e96b95529f2745f635190df5118a0cb44014
Summary:
Having the same queue for all three makes the dequeue code overly complicated
as it needs to keep track of the kind of request that needs to be dequeued.
Incidently, the previous code had a bug where request in "putback" would be
requeued at the end of the queue, even though there were at the beginning of it
if they all had the same priorities.
This is theory should also improve the dequeue performance when the queue has a
mix of blobs/tree requests, but I haven't measured.
Reviewed By: genevievehelsel
Differential Revision: D30560490
fbshipit-source-id: b27e5429105c07e5f9eab482c12e5699ca3413f7
Summary:
Since the background condition is before the actual prefetching of files,
specifying the background option would just glob files but not prefetch them
which is equivalent to prefetching all the trees.
Reviewed By: genevievehelsel
Differential Revision: D30618753
fbshipit-source-id: 5533b1c78d614342ac3341ce033795be3850750a
Summary:
It looks like a few scmstore changes landed with warnings (probably fixed higher up in the tree unification stack).
This change fixes those warnings.
Differential Revision: D30686092
fbshipit-source-id: d80625dea64f35683f815b58c83a3e5bb7cbdfa8
Summary:
When we remove a file from a sparse profile and commit the profile, it
should delete the file on disk. There's a bug where it doesn't actually delete
the file. This fixes it by passing the correct commit parents to the refresh
function.
Reviewed By: andll
Differential Revision: D30683677
fbshipit-source-id: 7f012faa99975d8270209f2962e7f9236890daed
Summary:
Use `populate_missing_vertexes_for_add_heads` (added by D27630093 (f138b012e9)) to avoid
excessive lookups for non-master ids that remain in non-master. The function
was used in two other `flush` cases, but missed the id reassignment case. It
works basically by using the "discovery" logic to quickly rule out what's
missing and what's present (ex. if a root is missing in the server graph, then
all descendants of the root is missing).
Reviewed By: andll
Differential Revision: D30700451
fbshipit-source-id: 1f1cd88399dbffd4af75083fef1f3e363a5c60fe
Summary: During reassigning non-master ids, it might trigger too many remote lookups.
Reviewed By: StanislavGlebik
Differential Revision: D30700452
fbshipit-source-id: 2483335e466c3de8a362f7b6a15fc4ba9e2693be
Summary:
Support octopus merge defined in the following format:
- The revlog flag has `1 << 12` set.
- Extra `stepparents` is set to `hexnode1,hexnode2,...` format.
This is mainly used to support revlog from stream clone.
Reviewed By: StanislavGlebik
Differential Revision: D30686450
fbshipit-source-id: d5aa2f18a02f5f0d7aa033210fb4f79b729c0d26
Summary:
Extend the struct so we can support more than 2 parents.
The size of the sturct is now 16 bytes, from 8 bytes. This might have some
performance overhead.
Not using `Box<[u8]>` because that will make the struct 24 bytes.
Reviewed By: StanislavGlebik
Differential Revision: D30686451
fbshipit-source-id: c0f8d0472c7e578f34d771dacecffc91585650c3
Summary:
In `vertex_id_with_max_group(name, group)`, if `group` is master and the `name`
exists in the non-master group, then there is no need to lookup remotely because
a same name (vertex) cannot be present in both master and non-master group. In
that case, just return that the `name` does not exist in the master group.
Reviewed By: StanislavGlebik
Differential Revision: D30699215
fbshipit-source-id: 5170abe719aa7cc31533912e18bc0e21f133e1f4
Summary:
Added a test about excessive remote lookups when flush() reassigns vertexes
from non-master to master.
Reviewed By: StanislavGlebik
Differential Revision: D30699214
fbshipit-source-id: 0547707764855ab9a563178740612b54df4a5fc9
Summary: They are used to narrow down issues related to S242328.
Reviewed By: StanislavGlebik
Differential Revision: D30699216
fbshipit-source-id: 28f4f0bfadadb2dea5510878168c2d7b47a8641c
Summary: Split out the request ID, URL, and HTTP method from `RequestContext` into a new `RequestInfo` struct, which can be cheaply cloned and included in the response returned to the caller. This enables the caller to correlate requests and responses, which is useful when working with many concurrent requests.
Reviewed By: DurhamG
Differential Revision: D30650365
fbshipit-source-id: 68efedcf852c91387450443ebe46062809633f10
Summary:
Make it possible to call the CheckIntegrity APIs from Python such as:
In [1]: cl.inner.checkuniversalids()
Out[1]: []
In [2]: cl.inner.checksegments()
Out[2]: []
In [3]: cl.inner.checkisomorphicgraph(cl.inner, cl.dageval(lambda: heads(mastergroup())))
# take a while
Out[3]: []
Reviewed By: andll
Differential Revision: D30682536
fbshipit-source-id: 23f280bf261def3d20d5f7dc15a48c2fc2d79d77
Summary: This makes other crates easier to implement CheckIntegrity.
Reviewed By: andll
Differential Revision: D30682540
fbshipit-source-id: 4333f37fa7bafe55a8bee9f149b2f23a463c51af
Summary:
Makes the revlog index provides dummy graph integrity checks so it can
be used as a generic object in the Python bindings.
Reviewed By: andll
Differential Revision: D30682542
fbshipit-source-id: 25c6e8640de46188d7bf45a927e11e0779a8ad40
Summary: Make it possible to check a graph against a reference graph.
Reviewed By: andll
Differential Revision: D30682539
fbshipit-source-id: 57db952dcda5656ff6000e9961448c9b64afbaf0
Summary: Make it possible to check segment integrity.
Reviewed By: andll
Differential Revision: D30644243
fbshipit-source-id: 24bb0c8c8c9394d688e3e9320e59268bc2a4ed3f
Summary: Make it possible to check universal ids externally.
Reviewed By: andll
Differential Revision: D30644242
fbshipit-source-id: f312ff59dbdf68e57c5249d57c5d44da0b10e398
Summary: This will be used to verify graph integrity later.
Reviewed By: andll
Differential Revision: D30644244
fbshipit-source-id: 0d22b70121da37c411adf17200a6c752fefa80ad
Summary: This breaks all use of `hg sparse`, because `.hg*` cannot be matched.
Reviewed By: mitrandir77
Differential Revision: D30666349
fbshipit-source-id: c06d1b798a57490f2e5560f178a2839ae5425146
Summary: We've got multiple manifold parameters now, two of which are Option<i64>, so lets create a struct to name them
Reviewed By: HarveyHunt
Differential Revision: D30305462
fbshipit-source-id: 44eee00d478e4485d074a14fcccec2f0f9572ecd
Summary:
This allows to quickly identify the program that emitted the error.
Per user feedback: https://fb.workplace.com/groups/clifoundation/posts/433922134631466
Reviewed By: StanislavGlebik
Differential Revision: D30604611
fbshipit-source-id: 712bc9f466c5a7b5c97a1b83a10fbe277341a300
Summary:
The mockall crate's `automock` attribute previously created nondeterministic output, which leads to frequent random "Found possibly newer version of crate" failures in Buck builds that involve cache.
The affected trait in Conveyor is:
https://www.internalfb.com/code/fbsource/[4753807291f7275a061d67cead04ea12e7b38ae2]/fbcode/conveyor/common/just_knobs/src/lib.rs?lines=13-23
which has a method with two lifetime parameters. Mockall's generated code shuffled them in random order due to emitting the lifetimes in HashSet order. The generated code would randomly contain one of these two types:
`Box<dyn for<'b, 'a> FnMut(&str, Option<&'a str>, Option<&'b str>) -> Result<bool> + Send>`
`Box<dyn for<'a, 'b> FnMut(&str, Option<&'a str>, Option<&'b str>) -> Result<bool> + Send>`
Reviewed By: jsgf
Differential Revision: D30656936
fbshipit-source-id: c1a251774333d7a4001a7492c1995efd84ff22e5
Summary: Adds an option to print the path to the eden log file. Similar to `eden pid`, this can be used for shell one-liners.
Reviewed By: chadaustin
Differential Revision: D30558294
fbshipit-source-id: ca70addaef2093e10f0321bae0cff3b1bfc7dc75
Summary: `eden debug log --upload` fits in better with the format of the other cli tools (rather than `eden debug log upload`)
Differential Revision: D30557691
fbshipit-source-id: 32e47e1487703560f2adb5f0f79f1002d29eea93
Summary:
In a previous diff we made sparse matchers become union matchers, since
they are a collection of each individual sparse profiles matcher. In order to
maintain the performance benefits of having sparse computations run on
non-python matchers, we need to update the matcher extractor to support union
matchers.
Reviewed By: quark-zju
Differential Revision: D30588256
fbshipit-source-id: 15014be844e1d713e19ae8f2959d947516b4e3c7
Summary:
We were copy/pasting metadata.get("version", "1") everywhere. Let's
make it a helper function.
Differential Revision: D30586162
fbshipit-source-id: ff6a9706f1970f84ffeb7de0e1362c3ba507fc00
Summary:
Sparse profiles should be roughly scoped around the files needed to
work on a certain product. If an engineer needs to work on multiple products
they should be able to enable multiple profiles.
Previously, multiple v2 profiles would be combined into an ordered list of
include/exclude rules, which meant that profiles enabled later could exclude
files included by the earlier profiles.
To fix this, let's treat each profile separately and create a matcher for each.
We then combine these into a union matcher, which means we're guaranteed to have
all the files that each profile specifies.
Differential Revision: D30586161
fbshipit-source-id: 2e04cfdba670ffce381a7c041706f315775ad7b0
Summary:
In a future diff we'll process sparse profiles differently at the
matcher creation level. To do so we need to expose the profile object to that
layer. Let's do it by storing the profile instead of just the profile name.
Differential Revision: D30586163
fbshipit-source-id: d90343b4101c43fbd838512289362aca7c3f816a
Summary:
With lazy changelog, it is possible that `vertex_name` is unable to translate
an id to name in non-async context. Do not treat it as errors.
Differential Revision: D30615948
fbshipit-source-id: 4e7abd77c6eb116db00e25489685563b7cf78a9c
Summary: The segments is stored in the shared `.hg`, not in the local repo `.hg`.
Differential Revision: D30615949
fbshipit-source-id: 9d2b7c1ce245553a2df070b066429fbcead5d827
Summary:
Similarly to the previous diff, reducing the lock scope will improve
concurrency leading to higher performance in EdenFS.
Reviewed By: andll
Differential Revision: D30595787
fbshipit-source-id: 1d52e4a8d362f7e2e3e18c2a57a3ebb7628f549e
Summary:
Similarly to the previous diff, let's not hold any read/write locks when not
needed. This will improve concurrency of the code.
Reviewed By: andll
Differential Revision: D30595786
fbshipit-source-id: 6ea6c689e4deca713051a9f3611647334c528bc7
