Summary:
Currently if we failed to fetch the repo status we only see "Repo is marked as read-only: Failed to fetch repo lock status" error, which is not very informative. Example of the error in production: P385612782.
Let's log the error.
Reviewed By: krallin
Differential Revision: D27621996
fbshipit-source-id: 85d9f0fe39397759da1b51e197f9188761678715
Summary:
Add support for returning unhydrated draft commits if requested by the client using a config option 'wantsunhydratedcommits'
This is needed to support slow enabling it for some clients like OnDemand.
Reviewed By: StanislavGlebik
Differential Revision: D27621442
fbshipit-source-id: 672129c8bfcbcdb4cee3ba1b092dac16c0b1877d
Summary:
We already log file count, but file sizes is another useful piece of
information.
I evaluated two options - either do as I did in this diff or just change ScribeToScuba
logging python script to query scs to get file sizes. I opted for option #1
mainly because scs doesn't have a method to query file sizes for many files at once, and
querying one by one might be too expensive. We can add a method like that, but
that's a bit of a bigger change that I'd like.
Reviewed By: andll
Differential Revision: D27620507
fbshipit-source-id: 2618e60845bc293535b190d4da85df5667a7ab60
Summary: This API has moved in Python 3. This is only used for util.popen4 calls, which aren't super frequent, so it escaped notice until now.
Reviewed By: sfilipco
Differential Revision: D27603246
fbshipit-source-id: 9ee6af4f66380480fe6c1c736287231464d0b21e
Summary: We were accidentally not setting the TLS key here; this diff fixes it.
Reviewed By: quark-zju
Differential Revision: D27634276
fbshipit-source-id: 9aac3a34b6f6655059a8d3332eea8ba02d062651
Summary:
Usage is hidden under config experimental.nativerebase
When config is set, we attempt to execute native rebase, and check if it has conflicts
If there are no conflicts, we convert native rebase result into Python actions
In case of conflicts, we currently just fallback to python merge
Reviewed By: DurhamG
Differential Revision: D27482909
fbshipit-source-id: 5705d372095f64ce0c6ee93c783e1c8dccf8b11a
Summary:
This diff contains proposal for basic rebase model for new checkout.
This diff introduces two types of file changes - basic no conflict Actions(that will replace CheckoutPlan actions) and ConflictAction for actions that contain conflicts
No conflict actions are produced when comparing two manifests
Conflict actions can be produced when rebasing manifests.
Eventually we will adjust checkout code to take either list of Actions or Conflict actions and perform corresponding operations on the file.
Rebase::rebase is used to generate list of conflict actions when rebasing.
Currently `Rebase::rebase` produces list of actions on top of 'base' commit.
Before doing actual checkout we will need to take into account what current commit is and adjust this list of actions accordingly (however, this adjustment will be trivial, it won't produce more conflicts)
After getting adjusted list of actions, checkout code can roll it out to fs and call merge drivers if needed.
In the future, if we will want to allow check in conflicts, we will need to adjust `Rebase::resolve_conflict` to take `ConflictAction` as input, instead of an Action.
Reviewed By: DurhamG
Differential Revision: D27028001
fbshipit-source-id: b02b7ad8030fcf146b53639d69e66fdc8f38a62d
Summary:
The nameset._set.iter() used by nameset.{fastdesc,fastdesc} do not know how to
prefetch. Just make them return `None` in this case. This addresses issues
where `log` can trigger one-by-one fetching.
hg log -r "limit(all(), 100000, 100000) & date('2021-04-01')"
The stack is a bit deep, it is:
# cmdutil.getlogrevs
if not revs:
...
# smartset.filteredset.__nonzero__
fast = None
candidates = [self.fastasc, self.fastdesc]
...
if fast is not None:
it = fast()
Basically filteredset calls into its _subset (nameset)'s fast{asc,desc},
which does not do prefetch properly and that cause filteredset.__nonzero__
to be slow.
Reviewed By: sfilipco
Differential Revision: D27634020
fbshipit-source-id: c3a44a1eb81c32f4dccb319ce42a15c56121c2ec
Summary:
Fixes MySQL syntax errors we've seen in some cases. No reason to call to the
database if we have no items to query for. It seems that empty queries can come
from the caching layer under certain configurations.
Reviewed By: krallin
Differential Revision: D27624798
fbshipit-source-id: 2febeff127f2fbdc739368ff1d1065c8f64723f8
Summary:
Remove the `DangerousOverride` mechanism for bookmarks traits and config.
Bookmarks overrides were only used to make unit test bookmarks use the same
metadata database as `SqlCounters`, which can now be done safely by getting the
metadata database from the factory.
Config overrides can be performed by the factory at repo construction.
Reviewed By: krallin
Differential Revision: D27424695
fbshipit-source-id: 0259da3abedb7ed4944fe945ba89800eea76ebff
Summary:
This dangerous override was being used to override
derived data config. Replace it with customizing the
config in the factory.
Reviewed By: krallin
Differential Revision: D27424696
fbshipit-source-id: 6dcf0c1397e217f09c0b82cf4700743c943f506f
Summary: This has been superseded by `RepoFactory`.
Reviewed By: krallin
Differential Revision: D27400617
fbshipit-source-id: e029df8be6cd2b7f3a5917050520b83bce5630e9
Summary:
Use `RepoFactory` to construct repositories in the walker.
The walker previously had special handling to allow repositories to
share metadata database and blobstore connections. This is now
implemented in `RepoFactory` itself.
Reviewed By: krallin
Differential Revision: D27400616
fbshipit-source-id: e16b6bdba624727977f4e58be64f8741b91500da
Summary: Add a way for users of `RepoFactory` to customize the blobstore that repos use.
Reviewed By: krallin
Differential Revision: D27400615
fbshipit-source-id: e3e515756c56dc78b8de8cf7b929109d05cec243
Summary:
Remove the dependency on blobrepo factory by defining a custom facet factory
for benchmark repositories.
Reviewed By: krallin
Differential Revision: D27400618
fbshipit-source-id: 626e19f09914545fb72053d91635635b2bfb6e51
Summary: Use `RepoFactory` to construct repositories in the LFS server.
Reviewed By: krallin
Differential Revision: D27363465
fbshipit-source-id: 09d5d32a133f166c6f308d56b2fb02f00031a179
Summary: Use `RepoFactory` to construct repositories in the microwave builder.
Reviewed By: krallin
Differential Revision: D27363468
fbshipit-source-id: 25bf2f7ee1ac0e52e1c6d4bda0c50ba67bc03110
Summary: Use the equivalent function from `repo_factory`.
Reviewed By: krallin
Differential Revision: D27363470
fbshipit-source-id: dce3cf843174caa2f9ef7083409e7935749be4cd
Summary:
This import is only used for the `ReadOnlyStorage` type, which is canonically
defined in `blobstore_factory`.
Reviewed By: krallin
Differential Revision: D27363474
fbshipit-source-id: 78fb1866d8a1223564357eea27ec0cdbe54fb5db
Summary:
This import is only used for the `ReadOnlyStorage` type, which is canonically
defined in `blobstore_factory`.
Reviewed By: krallin
Differential Revision: D27363466
fbshipit-source-id: 7cb1effcee6d39de92b471fecfde56724d24a6a4
Summary: Use `RepoFactory` to construct repositories for the hook tailer.
Reviewed By: krallin
Differential Revision: D27363472
fbshipit-source-id: 337664d7be317d2cfc35c7cd0f1f1230e39b6b43
Summary:
This import is only used for the `ReadOnlyStorage` type, which is canonically
defined in `blobstore_factory`.
Reviewed By: krallin
Differential Revision: D27363467
fbshipit-source-id: ed1388e661453e1b434c83af63c76da1eea1bce1
Summary: Use `RepoFactory` to construct repositories for all users of `cmdlib`.
Reviewed By: krallin
Differential Revision: D27363471
fbshipit-source-id: c9a483b41709fd90406c6600936671bf9ba61625
Summary:
Switch from `blobrepo_factory` to the new `RepoFactory` to construct `BlobRepo`
in `mononoke_api`.
The factory is part of the `MononokeEnvironment`, and is used to build all of the repos.
Reviewed By: krallin
Differential Revision: D27363473
fbshipit-source-id: 81345969e5899467f01d285c232a510b8edddb17
Summary:
To facilitate migration from `blobrepo_factory` to `repo_factory`, make common
types the same by re-exporting them from `repo_factory` in `blobrepo_factory`.
Reviewed By: ahornby
Differential Revision: D27323371
fbshipit-source-id: 9b0d98fe067de7905fc923d173ba8ae24eaa0d75
Summary:
Add a factory for building development and production repositories.
This factory can be re-used to build many repositories, and they will share
metadata database factories and blobstores if their configs match.
Similarly, the factory will only load redacted blobs once per metadata
database config, rather than once per repo.
Reviewed By: krallin
Differential Revision: D27323369
fbshipit-source-id: 951f7343af97f5e507b76fb822ad2e66f4d8f3bd
Summary: I found the Writer based zstd::Encoder api was doing a lot more allocations than the buffer based Compressor api, so switched as its both faster and better fit to the usecase.
Differential Revision: D27588448
fbshipit-source-id: ee8f72180045308a2e16709b9b5aa7bcf3b5cafd
Summary: Reduce amount of manual steps needed to restart a manual scrub by checkpointing where it has got to to a file.
Differential Revision: D27588450
fbshipit-source-id: cb0eda7d6ff57f3bb18a6669d38f5114ca9196b0
Summary: Here we further add prefetch-metadata support to prefetching profiles
Reviewed By: genevievehelsel
Differential Revision: D27568542
fbshipit-source-id: 64507125f47cf093c0133c82fcab941ed6495f32
Summary:
This is ... a stopgap :( There is probably some slow polling happening in
unbundle_future, and this causes us to fail to use our connection in time in
check_lock_repo...
Reviewed By: ahornby, StanislavGlebik
Differential Revision: D27620728
fbshipit-source-id: b747011405328b60419a99f0e5dbbaf64d53196a
Summary:
This one is a little bit trickier since we want to use Tokio inside a
Quickcheck function. That said, this is basically the expansion `tokio::main`
does, so we can simply use it.
Reviewed By: farnz
Differential Revision: D27619146
fbshipit-source-id: 1e3ea2d119913900d9b55c0a6d33de8a6ed5781c
Summary:
I'd like to just get rid of that library since it's one more place where we
specify the Tokio version and that's a little annoying with the Tokio 1.x
update. Besides, this library is largely obsoleted by `#[fbinit::test]` and
`#[tokio::test]`.
Reviewed By: farnz
Differential Revision: D27619147
fbshipit-source-id: 4a316b81d882ea83c43bed05e873cabd2100b758
Summary:
The intention is that the packer decides what to pack and in what order, PackBlob provides the methods needed to do the packing as requested by a packer.
Change the API so that a packer cannot make mistakes
Reviewed By: ahornby
Differential Revision: D27476427
fbshipit-source-id: 7dd534302c62b2432a2aca474f49da8ab9cbef1a
Summary:
It is useful to have latency stats grouped by the shardmap and label to easily identify where the problem comes from if something is broken.
This diff switches a single histogram used for all the MySQL use-cases into a set of histograms: one per `shardmap:label`. Ans also makes the histograms a bit smaller as we don't actually have such big numbers as 10s per conn/query.
There is only one case when the histogram is created per shard instead of a shardmap, it is `xdb.hgsql` DB with 9 shards. The reason why it happens it's because we connect to each shard as to an individual tier: https://fburl.com/diffusion/um8lt7cr.
{F582699426}
Reviewed By: farnz
Differential Revision: D27503833
fbshipit-source-id: 40c7eb64df7ae0694f63d3644231f240df8212ec
