Summary:
In production, all repos are instantiated roughly the same time so all reload
processes are started roughly the same time. Reload makes a bunch of requests
and could potentially cause load. Jitter spreads out the load of the reload.
Avoiding the load spike will make overall server behavior more predictable.
Reviewed By: krallin
Differential Revision: D27280117
fbshipit-source-id: 0727af2e7f231a5b6c948424022788a8e7071f82
Summary:
We would like to distribute the load of update process when many repositories
have Segmented Changelog enabled. Without the jitter all enabled repositories
start their update at roughly the same time. The jitter smooth out the load and
reduces variance in the update process.
Reviewed By: krallin
Differential Revision: D27280118
fbshipit-source-id: 41ad83b09700da1ef70c09dd5d284977e53a95a2
Summary:
`build_from_scratch` only called in `run_with_idmap_version` so we can inline
the code so that the seed process reads better.
This function used to be used as a shortcut towards getting a built dag but now
we prefer to fetch the dags from the store that the seeder writes to.
Reviewed By: krallin
Differential Revision: D27210036
fbshipit-source-id: 0b31ff1126a0f4904578da333cf6d34d69b2782c
Summary:
Removing the last callsite for SegmentedChangelogBuilder means that
the whole class goes away.
Reviewed By: krallin
Differential Revision: D27208339
fbshipit-source-id: 006aa91ed9656e4c33b082cbed82f9a497b0d267
Summary:
We are removing SegmentedChangelogBuilder.
Remove the last uses of Builder in the tests module.
Reviewed By: krallin
Differential Revision: D27208341
fbshipit-source-id: 00f1aaa2376ee5d68dbf7c1256b312cfe0b96d86
Summary:
Any functions that returns SegmentedChangelog is a valid argument for
reloading.
Reviewed By: krallin
Differential Revision: D27202520
fbshipit-source-id: fe903c6be4646c8ec98058d1a025829268c36619
Summary:
PeriodReloading is not fundamentally tied to the Manager. A future change will
update the load function.
Reviewed By: krallin
Differential Revision: D27202524
fbshipit-source-id: a0e4b08cb8605d071d5f30be8c3054f75321aa9c
Summary:
Let's look at these test from a higher perspective. Right now the tests use
internal APIs because not all components were ready when they were added. We
now have components for all the parts of the lifecycle so we can set up tests
in the same form that we would set up the production workflow.
This simplifies the API structure of the components since they can be catered
to one workflow.
Reviewed By: krallin
Differential Revision: D27202530
fbshipit-source-id: 6ec10a0b1ae49da13cfbe803e120a4e754b35fc7
Summary:
The broad goal is to get rid of SegmentedChangelogBuilder.
We will have a new constructor for Seeder, one that uses non segmented_changelog
dependencies as input.
Reviewed By: krallin
Differential Revision: D27202523
fbshipit-source-id: d420507502925d4440d5c3058efef0a4d2dbe895
Summary:
For tests that don't care about the bookmarks specifically, we want to use the
default bookmark name that we defined in BOOKMARK_NAME.
FWIW, it makes sense for this bookmark to be set by blobrepo even... at least
the fixtures. They set a bookmark and it makes sense for us to have a reference
to the bookmark that they set. Something to think about.
Reviewed By: krallin
Differential Revision: D27202522
fbshipit-source-id: 7615e4978dded491dd04ae44ce0b85134a252feb
Summary:
This gets rid of the odd builder for the Seeder.
We can get into design discussions with this one. What is struct and what is
function? For real structures that provide some behavior, I prefer to put
dependencies in owned data. Things that are part of the request go into
function parameters. In mononoke, RepositoryId is the common exception.
Anyway, IdMapVersion is part of the request for seeding. It is useful to have
that as a parameter when starting the seeder.
Reviewed By: krallin
Differential Revision: D27202528
fbshipit-source-id: a67b33493b20d2813fd0a144b9bb7f4510635ae8
Summary:
With the mix of external pager and progress suspension, the progress might
be enabled by accident:
# pager: disable forever
disable_progress(True)
# suspension
with progress.suspend()
...
# on __exit__, re-enables pager
Update the pager disabling logic be nested to avoid the potential issue.
Reviewed By: andll
Differential Revision: D27275016
fbshipit-source-id: 35ca7aef1890a981e6a1f0e5313b6a482ed43368
Summary:
Currently dropping progress bar panics if `__exit__` returns error
This happens, for example, when handling interrupts. Best course of actions just do not panic in this case
Reviewed By: sfilipco
Differential Revision: D27334897
fbshipit-source-id: c879fb14cfd4c16c0f9caede552129f8117defdc
Summary:
The progress total can be expensive to calculate for generatorset. Do not
calculate it. This will preserve laziness of certain sets (ex. generatorset
used by fastlog and average directory log).
Reviewed By: sfilipco
Differential Revision: D27327127
fbshipit-source-id: b0e3655e33b9e89ee2100941af18a769315f25bb
Summary:
Buffering the stream can provide suboptimal UX if the stream is slow.
Detect slow streams and avoid full buffering.
Reviewed By: sfilipco
Differential Revision: D27327128
fbshipit-source-id: a7b8037b7ba28fccc10661ffd15fd68f191d0048
Summary:
Remove use of dangerous_override from the repo client tests.
Previously this was used to override filestore config, so just use the existing
config override mechanism to set the filestore params this is generated from.
Reviewed By: ahornby
Differential Revision: D27169424
fbshipit-source-id: 7d17437f0e218d1cf19cc64d48e1efdd7012e927
Summary:
Remove use of dangerous_override in the pushrebase tests.
This was being used to ensure the bookmarks db in the test repo was shared with the mutable counters db. This is now directly possible by accessing the metadata database from the factory.
Reviewed By: ahornby
Differential Revision: D27169435
fbshipit-source-id: 1412231bdd9214bc869a3bfa7f63bf6c14db6836
Summary:
Remove uses of dangerous_override from derived data tests.
These were being used to override the derived data config or the lease object.
Extend the factory to allow customization of the config or the lease object,
and use that instead.
Reviewed By: StanislavGlebik
Differential Revision: D27169438
fbshipit-source-id: e8d0be248391d02bb054e19fdb9a90005db09c84
Summary:
Remove uses of dangerous_override in the commit rewriting tests.
Previously, the test was using dangerous_override to replace the bookmarks
attributes with ones that share a backing database with a `SqlMutableCounters`
instance.
With TestRepoFactory, all attributes share one metadata database. Obtain that
from the factory and use it to initialize the `SqlMutableCounters` instance.
Reviewed By: StanislavGlebik
Differential Revision: D27169429
fbshipit-source-id: 3c1b285db38a96deca7029d37e6692cb49356d31
Summary:
Remove dangerous overrides used in the tests in the bookmarks crate.
The test was using dangerous overrides to change the blobstore, which is now
supported directly by the test repo factory.
It was also using dangerous overrides to override the bookmark update log to
reset its database so it looks like the log is empty. Instead, clear out the
bookmark update log database as part of the test.
Reviewed By: StanislavGlebik
Differential Revision: D27169426
fbshipit-source-id: 64e1e89e31f62dcb585741ea728ebbe45f60fd38
Summary: This has been superseded by `test_repo_factory::TestRepoFactory`.
Reviewed By: StanislavGlebik
Differential Revision: D27169434
fbshipit-source-id: 97fcd400c5e3c6e8f86c9acfc5e979909e2eda31
Summary: Use the test factory for the remaining existing tests.
Reviewed By: StanislavGlebik
Differential Revision: D27169443
fbshipit-source-id: 00d62d7794b66f5d3b053e8079f09f2532d757e7
Summary: Use the test factory for existing bookmarks and pushrebase tests.
Reviewed By: StanislavGlebik
Differential Revision: D27169430
fbshipit-source-id: 65a7c87bc37cfa2b3b42873bc733cec177d8c1b0
Summary: Use the test factory for test fixtures that are used in many existing tests.
Reviewed By: StanislavGlebik
Differential Revision: D27169432
fbshipit-source-id: bb3cbfa95b330cf6572d1009c507271a7b008dec
Summary: Use the test factory for existing lfs_server tests.
Reviewed By: StanislavGlebik
Differential Revision: D27169427
fbshipit-source-id: 004867780ad6a41c3b17963006a7d3b0e5b82113
Summary: Use the test factory for existing commit_rewriting tests.
Reviewed By: StanislavGlebik
Differential Revision: D27169442
fbshipit-source-id: df2447b2b6423d172e684d7e702752ad717a2a4b
Summary: Use the test factory for existing repo_client and repo_import tests.
Reviewed By: StanislavGlebik
Differential Revision: D27169425
fbshipit-source-id: 2d0c34f129447232cec8faee42056d83613de179
Summary: Use the test factory for existing hooks and mapping tests.
Reviewed By: StanislavGlebik
Differential Revision: D27169436
fbshipit-source-id: f814e462bb2244e117c83bd355042016f5bfde8e
Summary: Use the test factory for existing mononoke_api tests.
Reviewed By: StanislavGlebik
Differential Revision: D27169444
fbshipit-source-id: d940bfa494dfe89fdb891d5248b73ba764f74faf
Summary: Use the test factory for existing derived data tests.
Reviewed By: farnz
Differential Revision: D27169433
fbshipit-source-id: b9b57a97f73aeb639162c359ab60c6fd92da14a8
Summary:
Create a factory that can be used to build repositories in tests.
The test repo factory will be kept in a separate crate to the production repo factory, so that it can depend on a smaller set of dependencies: just those needed for in-memory test repos. This should eventually help make compilation speeds faster for tests.
A notable difference between the test repos produced by this factory and the ones produced by `blobrepo_factory` is that the new repos share the in-memory metadata database. This is closer to what we use in production, and in a couple of places it is relied upon and existing tests must use `dangerous_override` to make it happen.
Reviewed By: ahornby
Differential Revision: D27169441
fbshipit-source-id: 82541a2ae71746f5e3b1a2a8a19c46bf29dd261c
Summary:
Convert `BlobRepo` to a `facet::container`. This will allow it to be built
from an appropriate facet factory.
This only changes the definition of the structure: we still use
`blobrepo_factory` to construct it. The main difference is in the types
of the attributes, which change from `Arc<dyn Trait>` to
`Arc<dyn Trait + Send + Sync + 'static`>, specified by the `ArcTrait` alias
generated by the `#[facet::facet]` macro.
Reviewed By: StanislavGlebik
Differential Revision: D27169437
fbshipit-source-id: 3496b6ee2f0d1e72a36c9e9eb9bd3d0bb7beba8b
Summary: To prepare for making `RepoBlobstore` a facet, convert it to a newtype wrapper.
Reviewed By: ahornby
Differential Revision: D27169439
fbshipit-source-id: ceefe307e962c03c3b89be660b5b6c18d79acf3e
Summary: We have support for backup-repo-id, but tw blobimport doesn't have id and have source repo name to use. Let's add support similar to other repo-id/source-repo-id etc.
Reviewed By: StanislavGlebik
Differential Revision: D27325583
fbshipit-source-id: 44b5ec7f99005355b8eaa4c066cb7168ec858049
Summary:
I'm trying to track down an issue in SQLBlob construction, where it takes over 3 wall-clock seconds to construct.
This is indicative of a missing `blocking` annotation somewhere; to make it easier to add the correct annotation, switch construction to modern futures and `async`
Reviewed By: krallin
Differential Revision: D27275801
fbshipit-source-id: 2b516b4eca7143e4be17c50c6542a9da601d6ac6
Summary:
The pack key needs to be removed using unlink() after target keys are pointed to it so that old unused packs can be GCed. e.g.
packer run 1: keys A and B are packed to P1
packer run 2: keys A and C are packed to P2
packer run 3: keys B and D are packed to P3
If we don't unlink P1 during run1 then GC can't collect the old pack after run 3 as the key will keep it live
Reviewed By: farnz
Differential Revision: D27188108
fbshipit-source-id: bbe22a011abbf85370a8c548413401ffb54b16b6
