Summary:
Add an endpoint to provide repo configuration information, such as whether
segmented changelog is supported by the server or not. This helps the client
make decisions without hitting actual (expensive) endpoints and distinguishing
from unrelated server errors. It would allow us to remove error-prone
client-side config deciding whether to use segment clone.
Reviewed By: krallin
Differential Revision: D30831346
fbshipit-source-id: 872e20a32879e075c75481f622b2a49000059d04
Summary:
In a future diff, we want an endpoint to test if segmented changelog is
supported for a repo without doing any real computation using segmented
chagnelog. This would be useful for the client to decide whether it can
use segmented changelog clone or not, instead of relying on fragile
per-repo configuration.
Reviewed By: farnz
Differential Revision: D30825920
fbshipit-source-id: 16dc5bf762da2d2b9cd808c129e1830285023f3d
Summary:
It's nice to have these functions to open source and target repos.
Previously we always had to get repo id first, and then call
open_repo_internal_with_repo_id
Reviewed By: yancouto
Differential Revision: D30866314
fbshipit-source-id: dd74822da755de232f4701f8523088e0bb612cb9
Summary:
D30829928 (fd03bff2e2) updated some of Mononoke's integration tests to take into
account whitespace changes. However, it removed the globs from some parts of
the tests.
As the assigned port changes on each test run, the globs are required. Add them
back in again, as well as fix up some whitespace in a test.
Reviewed By: markbt
Differential Revision: D30866884
fbshipit-source-id: 1557eee2143a2459a6412b8649e7e3dce5a607a4
Summary:
It's nice to have that can quickly count and print stats about a commit. I'm
using it now to understand performance of derived data.
Reviewed By: ahornby
Differential Revision: D30865267
fbshipit-source-id: 26b91c3c05a1c417015b5be228796589348bf064
Summary:
`rust_include_srcs` is supported on `thrift_library` as a way of including other Rust code in the generated crate, generally used to implement other traits on the generated types.
Adding support for this in autocargo by copying these files into the output dir and making sure their option is specified to the thrift compiler
Reviewed By: ahornby
Differential Revision: D30789835
fbshipit-source-id: 325cb59fdf85324dccfff20a559802c11816769f
Summary:
Add impls for Layer for Box/Arc<L: Layer> and <dyn Layer>. Also a pile of other
updates in git which haven't been published to crates.io yet, including proper
level filtering of trace events being fed into log.
Reviewed By: dtolnay
Differential Revision: D30829927
fbshipit-source-id: c01c9369222df2af663e8f8bf59ea78ee12f7866
Summary:
Bump all the versions on crates.io to highest to make migration to github
versions in next diff work.
Reviewed By: dtolnay
Differential Revision: D30829928
fbshipit-source-id: 09567c26f275b3b1806bf8fd05417e91f04ba2ef
Summary:
We don't need to pass the bubble id to the server, it can find it from the changeset id.
This fixes a TODO I added previously, and should make the `restore` command complete.
Reviewed By: ahornby
Differential Revision: D30609423
fbshipit-source-id: d1c8eb0e0556069fa408520a0aea91a0f865fbe1
Summary:
This diff adds an endpoint `/download/file` that allows to download a file given an upload token.
This will be used for snapshots, as we need to download the snapshot changes, and there's no way to do that right now.
Other options, and why I didn't do them:
- Using the existing `/files` endpoint: Not possible, as it needs hg filenodes and we don't have those.
- Returning the file contents in the fetch_snapshot request: Might make the response too big
- Returning just a single Bytes instead of a stream: I thought streaming would be preferred, and more future proof. In the stack I still put everything in memory in the client, but maybe in the future it should be possible to stream it directly to the file. I'm happy to remove if preferred, though.
Reviewed By: StanislavGlebik
Differential Revision: D30582411
fbshipit-source-id: f9423bc42867402d380e831bc45d3ce3b3825434
Summary:
The previous version had two issues:
1. It's UB to cast uninit away as it may be actually uninitialized.
2. Because of the cast, the buffer was not actually written nor advanced after written to, causing the caller to think nothing was read.
https://docs.rs/tokio/1.11.0/tokio/io/struct.ReadBuf.html
Reviewed By: dtolnay
Differential Revision: D30823808
fbshipit-source-id: d5f67e4c03f1d63f2241421dd35082ee96b5afd8
Summary: The new `EdenApiHandler` framework for defining EdenAPI endpoints provides a common place where responses are encoded. This diff adds automatic content compression at this point, using the received `Accept-Encoding` header from the request to determine what compression, if any, should be used. As a result, all endpoints that implement `EdenApiHandler` will get compression for free.
Reviewed By: yancouto
Differential Revision: D30553242
fbshipit-source-id: 9eda54cbf81dd1e03abec769744c96b16fad64ea
Summary:
We have two mode of deriving data: the "normal" way and using backfilling.
Backfilling is different from "normal" mode in that it derives a few commits at
once, and saves them all to blobstore at once.
Backfilling mode seemed to have helped us when we need to derive a lot of data
(e.g. backfill the whole repo). But
a) We don't know how much it helps, and we don't know if it depends on the repo
b) We don't know if it helps when we derive data for newly landed commits (i.e.
we use "backfill" mode in derived data tailer to derive data for latest public
commits)
So this diff adds a bit of logging to a separate scuba table so that we can get
an idea about things like:
1) How long does it take to derive a stack of commits?
2) Where do we spend most of the time (e.g. deriving, saving the blobs, saving
the mapping).
Reviewed By: mitrandir77
Differential Revision: D30805504
fbshipit-source-id: d82c905cafa87459990d74769a0dddcc91fac174
Summary:
It allows us to do 3 things:
1) Remove derive function
2) Add support for backfill mode so that we can compare perf with and without
it
3) Use all derived data types, and not just 3 of them
Reviewed By: krallin
Differential Revision: D30804258
fbshipit-source-id: 604723a3d845a60cfd94b4e090a121f5b5191536
Summary: This command can be useful to split a large bonsai commit into a smaller one
Reviewed By: mitrandir77
Differential Revision: D30776789
fbshipit-source-id: dc56d7c51eb0e9e0988dcba868c6008ebf488927
Summary:
While we don't really need it, creation mononoke matches fail if they are not
present. Let's just enable it here - it's not a bad thing to initialize them
Reviewed By: mitrandir77
Differential Revision: D30780463
fbshipit-source-id: c4199c6711ae7bd9641e9f51643b94d020051dbd
Summary: Implement serverside graph endpoint for fetching the mapping of commits to commit parents for the missing segment of a commit graph. This implementation uses the find_commits_to_send method from the get_bundle_response library. What may be missing from pull and the old bundle protocol now is mutation markers.
Reviewed By: yancouto
Differential Revision: D30485672
fbshipit-source-id: ba3a30d9e482d60831cbe7a8e89f20dab947d9a1
Summary:
Since the find_commits_to_send method was added, common is already a
hashset not a vector, so it doesn't needed to be converted to a hashset.
Reviewed By: quark-zju
Differential Revision: D30622028
fbshipit-source-id: e5d1b6c60115d13c906b25142043652ba9e89d70
Summary:
Currently there are two things preventing us from running add_sync_target
on existing target:
* already existing bookmark
* already existing config
Both need to be deleted to create new target. This diff removes the second
one to simplify code and make it easier to recreate the target (it's easy to
forget about manually removing the config as they otherwise don't need
human interventions).
Reviewed By: StanislavGlebik
Differential Revision: D30767613
fbshipit-source-id: f951c0e1ef9bde69d805dc911331fcdb220123f2
Summary:
Update the `Filenodes` trait so that it doesn't require the repository id to be
passed in every method invocation. In practice a filenodes instance can only
be used for a single repo, so it is safer for the implementation to store the
repository id.
At the same time, update the trait to use new futures and async-trait.
Reviewed By: yancouto
Differential Revision: D30729630
fbshipit-source-id: a1f80a299d9b0a99ddb267d1f7093f27cf21f1af
Summary:
Make the derivation process for mercurial changesets and manifests not depend
on `BlobRepo`, but instead use the repo attribute (`RepoBlobstore`) directly.
This will allow us to migrate to using `DerivedDataManager` in preparation
for removing `BlobRepo` from derivation entirely.
Reviewed By: yancouto
Differential Revision: D30729629
fbshipit-source-id: cf478ffb97a919c78c7e6e574580218539eb0fdf
Summary:
Make the derivation process for blame and deleted files manifest not depend
on `BlobRepo`, but instead use the repo attribute (`RepoBlobstore`) directly.
This will allow us to migrate to using `DerivedDataManager` in preparation
for removing `BlobRepo` from derivation entirely.
A `BlobRepo` reference is still needed at the moment for derivation of the
unodes that these depend on. That will be removed when `DerivedDataManager`
takes over co-ordination of derivation.
Reviewed By: yancouto
Differential Revision: D30729628
fbshipit-source-id: 4504abbe63c9bf036d69cb4341c75b13061fae18
Summary:
Make the derivation process for fsnodes and skeleton manifests not depend on
`BlobRepo`, but instead take the `DerivedDataManager` from the `BlobRepo` and
use that instead. This is in preparation for removing `BlobRepo` from
derivation entirely.
Reviewed By: yancouto
Differential Revision: D30301855
fbshipit-source-id: a2ed1639526aad9ddbe8429988043f0499f7629c
Summary:
Make the derivation process for unodes not depend on `BlobRepo`, but instead
take the `DerivedDataManager` from the `BlobRepo` and use that instead. This
is in preparation for removing `BlobRepo` from derivation entirely.
Reviewed By: yancouto
Differential Revision: D30300408
fbshipit-source-id: c35e9e21366de74338f453aaf6be476e7305556d
Summary:
The derived data manager also has a reference to the repo blobstore. This must
also be overridden when we override the blobstore.
Reviewed By: yancouto
Differential Revision: D30738354
fbshipit-source-id: b0e16ef810c5244cd056a3c9e5b9ceaaddb5ecea
Summary:
I want to use `ReplicaFirst` read connection type since `ReplicaOnly` is a bit too restrictive.
We've had 2 MySQL SEVs this year when all the replicas went down crashing our services despite the primary instance working normally.
There was also a case when I've deleted too much rows at once and all replicas went down due to replication lag (I know better now)
RFC
- Yay or Nay?
- Should I expand `ReadConnectionType` to mirror all options of `InstanceRequirement`?
- Perhaps it's worth moving it into the `common/rust/shed/sql` crate?
I kept cleaning up all the usages out of this diff to keep the changes minimal for RFC
Differential Revision: D30574326
fbshipit-source-id: 1462b238305d47557372afe7763096c53df55f10
Summary:
Segmented changelog seeder spends a significant chunk of time fetching
changesets. By saving them to file we can make reseeding significantly faster.
Reviewed By: farnz
Differential Revision: D30765374
fbshipit-source-id: 0e6adf12e334ad70486145173ae87c810880988a
Summary:
In backfill_derived_data we had a way to prefetch a lot of commits at once, so
that backfill_derived_data doesn't have to do it on every startup.
I'd like to use the same functionality in segmented changelog seeder, so let's
move it to the separate binary.
Reviewed By: mitrandir77, farnz
Differential Revision: D30765375
fbshipit-source-id: f6930965b270cbaae95c3ac4390b3d367eaab338
Summary:
~~Also enable the `derive` feature so it isn't necessary to separately
depend on `strum_macros`.~~
This turns out to break a lot.
Reviewed By: dtolnay
Differential Revision: D30709976
fbshipit-source-id: a9181070b8d7a8489eebc9e94fa24f334cd383d5
Summary: Looks like it was lost during the last refactoring, let's add it back.
Reviewed By: farnz
Differential Revision: D30728456
fbshipit-source-id: 20c638b3c5a8664f2367f871cd29a793fd897de3
Summary:
Like it says in the title, this updates us to use Daemonize 0.5, though from
Github and not Crates.io, because it hasn't been released to the latter yet.
The main motivation here is to pull in
https://github.com/knsd/daemonize/pull/39 to avoid leaking PID files to
children of the daemon.
This required some changes in `hphp/hack/src/facebook/hh_decl` and `xplat/rust/mobium` since the way to
run code after daemonization has changed (and became more flexible).
Reviewed By: ndmitchell
Differential Revision: D30694946
fbshipit-source-id: d99768febe449d7a079feec78ab8826d0e29f1ef
Summary:
At the moment when segmented changelog is updated and/or reseeded mononoke
servers can pick it up only once an hour (this is a current reload schedule)
or when mononoke server is restarted. However during production issues (see
attached task for an example) it would be great to have a way to force all
servers to reload segmented changelog.
This diff makes it possible to do so with a tunable. Once tunable changes its
value then monononoke servers almost immediately (subject to jitter) reload it.
This implementation adds a special loop that polls tunables value and reloads
if it changes. Note that in theory it could avoid polling and watch for configerator
changes instead, but it would be harder to implement and I decided that it's
not worth it.
Reviewed By: farnz
Differential Revision: D30725095
fbshipit-source-id: da90ea06715c4b763d0de61e5899dfda8ffe2067
Summary:
Manual component version update
Bump Schedule: https://www.internalfb.com/intern/msdk/bump/?schedule_fbid=342556550408072
Package: https://www.internalfb.com/intern/msdk/package/181247287328949/
Oncall Team: rust_foundation
NOTE: This build is expected to expire at 2022/09/01 09:14AM PDT
---------
New project source changes since last bump based on D30663071 (08e362a355e0a64a503f5073f57f927394696b8c at 2021/08/31 03:47AM -05):
| 2021/08/31 04:41AM -05 | generatedunixname89002005294178 | D30665384 | [MSDK] Update autocargo component on FBS:master |
| 2021/08/31 07:14PM PDT | kavoor | D30681642 | [autocargo] Make cxx-build match version of cxx |
| 2021/09/01 04:05PM BST | krallin | D30698095 | autocargo: include generated comment in OSS manifests |
---------
build-break (bot commits are not reviewed by a human)
Reviewed By: farnz
Differential Revision: D30717040
fbshipit-source-id: 2c1d09f0d51b6ff2e2636496cf22bcf781f22889
Summary: We've got multiple manifold parameters now, two of which are Option<i64>, so lets create a struct to name them
Reviewed By: HarveyHunt
Differential Revision: D30305462
fbshipit-source-id: 44eee00d478e4485d074a14fcccec2f0f9572ecd
Summary:
This allows to quickly identify the program that emitted the error.
Per user feedback: https://fb.workplace.com/groups/clifoundation/posts/433922134631466
Reviewed By: StanislavGlebik
Differential Revision: D30604611
fbshipit-source-id: 712bc9f466c5a7b5c97a1b83a10fbe277341a300
Summary:
The mockall crate's `automock` attribute previously created nondeterministic output, which leads to frequent random "Found possibly newer version of crate" failures in Buck builds that involve cache.
The affected trait in Conveyor is:
https://www.internalfb.com/code/fbsource/[4753807291f7275a061d67cead04ea12e7b38ae2]/fbcode/conveyor/common/just_knobs/src/lib.rs?lines=13-23
which has a method with two lifetime parameters. Mockall's generated code shuffled them in random order due to emitting the lifetimes in HashSet order. The generated code would randomly contain one of these two types:
`Box<dyn for<'b, 'a> FnMut(&str, Option<&'a str>, Option<&'b str>) -> Result<bool> + Send>`
`Box<dyn for<'a, 'b> FnMut(&str, Option<&'a str>, Option<&'b str>) -> Result<bool> + Send>`
Reviewed By: jsgf
Differential Revision: D30656936
fbshipit-source-id: c1a251774333d7a4001a7492c1995efd84ff22e5
Summary:
this library is a more general version of the `panic_unpack` lib I
made in fbcode. I made this library, its mit-apache licensed
Reviewed By: dtolnay
Differential Revision: D30607308
fbshipit-source-id: ee4fad3924fdae021753055cd3fd88c99cb99512
Summary:
I abandoned D30603353 in favor of this one because
its_cleaner
We don't need repo name in every hook, just 2 of them.
This will allow us to have predefined, named hook configs that are repo-agnostic. E.g. when repositories are similar, they could share one set of hook configs in configerator.
There were two hooks that had repo_name in configerator hook config: verify_integrity and verify_reviewed_by
Reviewed By: StanislavGlebik
Differential Revision: D30605229
fbshipit-source-id: c310b16b564808d0dc0909d21cc3521a57e06fad
Summary: Cleaned up all remaining usages of this deprecated API in CTP codebase
Differential Revision: D30517771
fbshipit-source-id: 6b2c7fb6c569bf5a928a7eec60fdd890baad312f
Summary:
Handling of mutable renames was incorrect for two reasons:
1) We didn't add an entry to history graph, so only a single changeset before
rename was returned. That was easy to "fix" (just add a new entry to history
graph), but...
2) ...all history operations now have to use a different path (the source of
the rename path).
To fix it let's track not just the changeset id, but also the path for the
given changeset id. Since the path can potentially be large I wrapped it into
Arc to avoid expensive clones.
Differential Revision: D30576342
fbshipit-source-id: a99f6269c34b0a0c626104ec47c9392f984328fb
Summary:
This config option was used to slowly roll out LFS for a repo.
However, it is no longer used and can therefore be removed.
Reviewed By: StanislavGlebik
Differential Revision: D30511880
fbshipit-source-id: 59fe5925cc203aa609488fdf8ea29e9ff65ee862
Summary:
This adds a new endpoint to EdenApi: `/:repo/snapshot`.
It fetches information about a snapshot (for now, parents and files).
This will be used by the `hg snapshot restore` command.
- Parent will be used to do a normal "hg update" to it
- File changes will be used to apply the working state file changes on top of it.
Reviewed By: ahornby
Differential Revision: D30514854
fbshipit-source-id: b6c6930410cf15fe874eca1fce54314e5011512a
