sapling/eden/mononoke
Kostia Balytskyi c01294e8d6 backsyncer_cmd: use LiveCommitSyncConfig
Summary:
This diff migrates `backsyncer_cmd` (the thing that runs in the separate backsyncer job, as opposed to bakcsyncer, triggered from push-redirector) onto `LiveCommitSyncConfig`. Specifically, this means that on every iteration of the loop, which calls `backsync_latest` we reload `CommitSyncConfig` from configerator, build a new `CommitSyncer` from it, and then pass that `CommitSyncer` to `backsync_latest`.

One choice made here is to *not* create `CommitSyncer` on every iteration of the inner loop of `backsync_latest` and handle live configs outside. The reason for this is twofold:
- `backsync_latest` is called form `PushRedirector` methods, and `PushRedirector` is recreated on each `unbundle` using `LiveCommitSyncConfig`. That call provides an instance of `CommitSyncer` used to push-redirect a commit we want to backsync. It seems strictly incorrect to try and maybe use a different instance.
- because of some other consistency concerns (different jobs getting `CommitSyncConfig` updates at different times), any sync config change needs to go through the following loop:
  - lock the repo
  - land the change
  - wait some time, until all the possible queues (x-repo sync and backsync) are drained
  - unlock the repo
- this means that it's ok to have the config refreshed outside of `backsync_latest`

Reviewed By: farnz

Differential Revision: D22206992

fbshipit-source-id: 83206c3ebdcb2effad7b689597a4522f9fd8148a
2020-06-26 13:40:31 -07:00
..
backsyncer add newtype for CommitSyncConfigVersion 2020-06-26 02:45:26 -07:00
benchmark Switch Loadable and Storable interfaces to new-style futures 2020-06-25 08:45:37 -07:00
blobimport_lib move all mercurial content generation logic to blobrepo_hg 2020-06-22 07:29:19 -07:00
blobrepo Switch Blobstore to new-style futures 2020-06-26 03:54:42 -07:00
blobrepo_utils Switch Loadable and Storable interfaces to new-style futures 2020-06-25 08:45:37 -07:00
blobstore Support BlobstoreWithLink in Sqlblob 2020-06-26 03:54:42 -07:00
blobstore_sync_queue mononoke: blobstore_sync_queue: switch to insert rather than insert ignore 2020-06-10 19:29:41 -07:00
bonsai_git_mapping Switch Loadable and Storable interfaces to new-style futures 2020-06-25 08:45:37 -07:00
bonsai_globalrev_mapping Switch Loadable and Storable interfaces to new-style futures 2020-06-25 08:45:37 -07:00
bonsai_hg_mapping eden: remove unused Rust dependencies 2020-06-17 17:55:03 -07:00
bookmarks move DangerousOverride into a separate crate blobrepo_override 2020-06-22 07:29:19 -07:00
bulkops move all mercurial content generation logic to blobrepo_hg 2020-06-22 07:29:19 -07:00
cache_warmup Switch Loadable and Storable interfaces to new-style futures 2020-06-25 08:45:37 -07:00
changesets eden: remove unused Rust dependencies 2020-06-17 17:55:03 -07:00
cmdlib cmdlib: expose test-instance and local-configerator-path 2020-06-26 06:51:07 -07:00
cmds Switch Blobstore to new-style futures 2020-06-26 03:54:42 -07:00
commit_rewriting add newtype for CommitSyncConfigVersion 2020-06-26 02:45:26 -07:00
common Switch Blobstore to new-style futures 2020-06-26 03:54:42 -07:00
derived_data Switch Blobstore to new-style futures 2020-06-26 03:54:42 -07:00
edenapi_server mononoke: try to compress values if they above cachelib limit 2020-06-20 01:05:54 -07:00
fastreplay Switch Blobstore to new-style futures 2020-06-26 03:54:42 -07:00
filenodes eden: remove unused Rust dependencies 2020-06-17 17:55:03 -07:00
filestore Switch Blobstore to new-style futures 2020-06-26 03:54:42 -07:00
git Switch Blobstore to new-style futures 2020-06-26 03:54:42 -07:00
gotham_ext mononoke/lfs_server: support the client not having the data it wants to send us 2020-06-24 10:02:01 -07:00
hgcli mononoke: add Cargo.toml files for code previously depending on identity_ext 2020-06-10 18:16:04 -07:00
hgproto edenapi_server: rename the subtree endpoint to complete_trees 2020-06-15 13:40:44 -07:00
hook_tailer Switch Loadable and Storable interfaces to new-style futures 2020-06-25 08:45:37 -07:00
hooks Switch Loadable and Storable interfaces to new-style futures 2020-06-25 08:45:37 -07:00
lfs_import_lib autocargo: regenerate 2020-05-19 16:08:40 -07:00
lfs_protocol mononoke: try to compress values if they above cachelib limit 2020-06-20 01:05:54 -07:00
lfs_server Switch Blobstore to new-style futures 2020-06-26 03:54:42 -07:00
load_limiter mononoke/configerator structs: change the OSS folder where configerator structs are saved 2020-06-24 09:38:46 -07:00
manifest Switch Blobstore to new-style futures 2020-06-26 03:54:42 -07:00
megarepolib Switch Loadable and Storable interfaces to new-style futures 2020-06-25 08:45:37 -07:00
mercurial Switch Blobstore to new-style futures 2020-06-26 03:54:42 -07:00
metaconfig add newtype for CommitSyncConfigVersion 2020-06-26 02:45:26 -07:00
microwave Switch Blobstore to new-style futures 2020-06-26 03:54:42 -07:00
mononoke_api Switch Loadable and Storable interfaces to new-style futures 2020-06-25 08:45:37 -07:00
mononoke_types Switch Blobstore to new-style futures 2020-06-26 03:54:42 -07:00
mutable_counters admin: add a subcommand to manipulate mutable_counters 2020-04-16 03:19:44 -07:00
newfilenodes eden: manual dependency fixes 2020-06-17 17:55:04 -07:00
permission_checker mononoke: replace instances of whitelist/blacklist/blackhole 2020-06-10 19:29:30 -07:00
phases remove HgPhase type 2020-06-22 13:51:33 -07:00
pushrebase Switch Loadable and Storable interfaces to new-style futures 2020-06-25 08:45:37 -07:00
reachabilityindex Switch Blobstore to new-style futures 2020-06-26 03:54:42 -07:00
repo_client Switch Blobstore to new-style futures 2020-06-26 03:54:42 -07:00
repo_import mononoke/repo_import: Add gitimport functionality and integration test 2020-06-25 19:54:38 -07:00
revset BlobRepoHg extension trait. 2020-06-22 07:29:19 -07:00
scs_server/src rust/thrift: add an option to stop processing requests if client disconnected, take 2 2020-06-15 07:17:12 -07:00
segmented_changelog eden: manual dependency fixes 2020-06-17 17:55:04 -07:00
server cmdlib: expose test-instance and local-configerator-path 2020-06-26 06:51:07 -07:00
sshrelay Clean out unused deps from our TARGETS files 2020-04-14 03:38:11 -07:00
tests backsyncer_cmd: use LiveCommitSyncConfig 2020-06-26 13:40:31 -07:00
time_window_counter mononoke/rust-shed: apply new coding style convention that separates oss-only code 2020-05-07 04:52:16 -07:00
tunables mononoke/configerator structs: change the OSS folder where configerator structs are saved 2020-06-24 09:38:46 -07:00
unbundle_replay Switch Loadable and Storable interfaces to new-style futures 2020-06-25 08:45:37 -07:00
walker Switch Loadable and Storable interfaces to new-style futures 2020-06-25 08:45:37 -07:00
Cargo.toml Switch Blobstore to new-style futures 2020-06-26 03:54:42 -07:00
README.md mononoke: add README.md and the missing pieces for supporting cargo (#13) 2020-02-13 00:12:36 -08:00

Mononoke

Mononoke is a next-generation server for the Mercurial source control system, meant to scale up to accepting thousands of commits every hour across millions of files. It is primarily written in the Rust programming language.

Caveat Emptor

Mononoke is still in early stages of development. We are making it available now because we plan to start making references to it from our other open source projects.

The version that we provide on GitHub does not build yet.

This is because the code is exported verbatim from an internal repository at Facebook, and not all of the scaffolding from our internal repository can be easily extracted. The key areas where we need to shore things up are:

  • Full support for a standard cargo build.
  • Open source replacements for Facebook-internal services (blob store, logging etc).

The current goal is to get Mononoke working on Linux. Other Unix-like OSes may be supported in the future