sapling/eden/mononoke
Simon Farnsworth 103b3116d4 Use filenodes where available to speed up derived data for merges
Summary:
Fetching all history of both filenodes to see if there's common history either side of a merge is wasteful, and in some megarepo work is causing long delays deriving merge changesets.

Where we have already derived filenodes for a given merge's ancestors, we can go faster; we can use the linknodes to determine the older of the two filenodes, and fetch only history for the newer of the two.

This is imperfect for the import use case, since filenodes depend on hgchangesets, and the batching in use at the moment prefers to generate all derived data of a given type before moving onto another type, but it's an improvement for cases where some filenodes are already derived (e.g. due to import of a repo with a similar history).

Reviewed By: StanislavGlebik

Differential Revision: D28796253

fbshipit-source-id: 5384b5d2841844794a518c321dbf995891374d3a
2021-06-03 02:33:40 -07:00
..
admin thrift/lib/rust: update to Bytes 1.x 2021-05-20 09:44:41 -07:00
aliasverify thrift/lib/rust: update to Bytes 1.x 2021-05-20 09:44:41 -07:00
alpn rust: remove patch for async-compression 2021-05-05 12:20:34 -07:00
backfill_derived_data Do not alter rate limit inside derived data utils 2021-05-24 11:43:01 -07:00
benchmark_filestore thrift/lib/rust: update to Bytes 1.x 2021-05-20 09:44:41 -07:00
benchmarks mononoke: sync changeset implementation for megarepo 2021-05-13 10:04:21 -07:00
blobimport manifold/client: take IOBufShared data as input for writes 2021-05-19 09:57:20 -07:00
blobimport_lib thrift/lib/rust: update to Bytes 1.x 2021-05-20 09:44:41 -07:00
blobrepo Use filenodes where available to speed up derived data for merges 2021-06-03 02:33:40 -07:00
blobrepo_utils mononoke: sync changeset implementation for megarepo 2021-05-13 10:04:21 -07:00
blobstore mononoke: add InlineBase64 chunking to sqlblob 2021-06-01 17:24:45 -07:00
blobstore_healer thrift/lib/rust: update to Bytes 1.x 2021-05-20 09:44:41 -07:00
blobstore_sync_queue mononoke: sync changeset implementation for megarepo 2021-05-13 10:04:21 -07:00
bonsai_git_mapping mononoke: sync changeset implementation for megarepo 2021-05-13 10:04:21 -07:00
bonsai_globalrev_mapping thrift/lib/rust: update to Bytes 1.x 2021-05-20 09:44:41 -07:00
bonsai_hg_mapping thrift/lib/rust: update to Bytes 1.x 2021-05-20 09:44:41 -07:00
bonsai_svnrev_mapping thrift/lib/rust: update to Bytes 1.x 2021-05-20 09:44:41 -07:00
bookmarks thrift/lib/rust: update to Bytes 1.x 2021-05-20 09:44:41 -07:00
bulkops mononoke: sync changeset implementation for megarepo 2021-05-13 10:04:21 -07:00
cache_warmup mononoke: sync changeset implementation for megarepo 2021-05-13 10:04:21 -07:00
changesets thrift/lib/rust: update to Bytes 1.x 2021-05-20 09:44:41 -07:00
cmdlib mononoke: option for how far back to peek at blobstore queue when scrubbing 2021-05-21 05:21:18 -07:00
cmds Remove --sparse argument on skiplist build 2021-06-01 07:29:33 -07:00
commit_rewriting thrift/lib/rust: update to Bytes 1.x 2021-05-20 09:44:41 -07:00
common mononoke: allow deriving data in parallel for independent commits 2021-06-03 02:23:15 -07:00
derived_data mononoke: derive data in add_sync_target 2021-06-03 02:23:15 -07:00
edenapi_server mononoke: sync changeset implementation for megarepo 2021-05-13 10:04:21 -07:00
edenapi_service Daily common/rust/cargo_from_buck/bin/autocargo 2021-05-25 17:03:52 -07:00
fastreplay Update to Rust 1.52.1 2021-05-19 22:13:47 -07:00
filenodes rust: remove patch for async-compression 2021-05-05 12:20:34 -07:00
filestore thrift/lib/rust: update to Bytes 1.x 2021-05-20 09:44:41 -07:00
git histroy -> history 2021-06-02 10:26:41 -07:00
gotham_ext Daily common/rust/cargo_from_buck/bin/autocargo 2021-05-25 17:03:52 -07:00
hgcli add compression support 2021-05-20 10:22:56 -07:00
hgproto rust: remove patch for async-compression 2021-05-05 12:20:34 -07:00
hook_tailer mononoke: sync changeset implementation for megarepo 2021-05-13 10:04:21 -07:00
hooks Daily common/rust/cargo_from_buck/bin/autocargo 2021-05-25 17:03:52 -07:00
lfs_import thrift/lib/rust: update to Bytes 1.x 2021-05-20 09:44:41 -07:00
lfs_import_lib thrift/lib/rust: update to Bytes 1.x 2021-05-20 09:44:41 -07:00
lfs_protocol rust: remove patch for async-compression 2021-05-05 12:20:34 -07:00
lfs_server mononoke/lfs_server: include oid in ObjectCannotBeSynced error 2021-06-01 02:57:07 -07:00
load_limiter rust: remove patch for async-compression 2021-05-05 12:20:34 -07:00
manifest rust: remove patch for async-compression 2021-05-05 12:20:34 -07:00
megarepo_api mononoke: derive data in add_sync_target 2021-06-03 02:23:15 -07:00
mercurial thrift/lib/rust: update to Bytes 1.x 2021-05-20 09:44:41 -07:00
metaconfig mononoke: sample harder in the walker 2021-05-06 07:07:02 -07:00
microwave mononoke: sync changeset implementation for megarepo 2021-05-13 10:04:21 -07:00
mononoke_api thrift/lib/rust: update to Bytes 1.x 2021-05-20 09:44:41 -07:00
mononoke_api_hg thrift/lib/rust: update to Bytes 1.x 2021-05-20 09:44:41 -07:00
mononoke_hg_sync_job Daily common/rust/cargo_from_buck/bin/autocargo 2021-05-25 17:03:52 -07:00
mononoke_types blame_v2: add "skip past this change" metadata 2021-05-24 03:49:27 -07:00
mutable_counters mononoke: sync changeset implementation for megarepo 2021-05-13 10:04:21 -07:00
newfilenodes thrift/lib/rust: update to Bytes 1.x 2021-05-20 09:44:41 -07:00
observability rust: remove patch for async-compression 2021-05-05 12:20:34 -07:00
permission_checker option to enforce authN 2021-05-20 01:06:04 -07:00
phases thrift/lib/rust: update to Bytes 1.x 2021-05-20 09:44:41 -07:00
pushrebase mononoke: sync changeset implementation for megarepo 2021-05-13 10:04:21 -07:00
pushrebase_mutation_mapping rust: remove patch for async-compression 2021-05-05 12:20:34 -07:00
quiet_stream mononoke: sync changeset implementation for megarepo 2021-05-13 10:04:21 -07:00
reachabilityindex Reload skip lists periodically 2021-05-27 09:31:35 -07:00
regenerate_hg_filenodes rust: remove patch for async-compression 2021-05-05 12:20:34 -07:00
repo_attributes megarepo_api: introduce a MegarepoApi struct to rule them all 2021-05-12 12:00:20 -07:00
repo_client thrift/lib/rust: update to Bytes 1.x 2021-05-20 09:44:41 -07:00
repo_factory mononoke: add a simple test for sync changeset 2021-05-18 07:37:56 -07:00
repo_import reenable -debuginfo packages 2021-05-24 02:04:36 -07:00
revset mononoke: sync changeset implementation for megarepo 2021-05-13 10:04:21 -07:00
scs change the return format of poll requests to include the request result 2021-06-02 01:40:52 -07:00
scs_server use real worker instead of completing requests in SCS 2021-06-02 01:40:52 -07:00
segmented_changelog thrift/lib/rust: update to Bytes 1.x 2021-05-20 09:44:41 -07:00
server Daily common/rust/cargo_from_buck/bin/autocargo 2021-05-25 17:03:52 -07:00
sshrelay add compression support 2021-05-20 10:22:56 -07:00
statistics_collector thrift/lib/rust: update to Bytes 1.x 2021-05-20 09:44:41 -07:00
streaming_clone mononoke: sync changeset implementation for megarepo 2021-05-13 10:04:21 -07:00
tests use real worker instead of completing requests in SCS 2021-06-02 01:40:52 -07:00
time_window_counter rust: remove patch for async-compression 2021-05-05 12:20:34 -07:00
tunables mononoke: allow deriving data in parallel for independent commits 2021-06-03 02:23:15 -07:00
unbundle_replay thrift/lib/rust: update to Bytes 1.x 2021-05-20 09:44:41 -07:00
upload_globalrevs thrift/lib/rust: update to Bytes 1.x 2021-05-20 09:44:41 -07:00
walker mononoke: log ctime in walker packinfo logs 2021-06-01 10:59:55 -07:00
Cargo.toml introduce async_requests_worker 2021-05-27 02:40:11 -07:00
README.md mononoke: add README.md and the missing pieces for supporting cargo (#13) 2020-02-13 00:12:36 -08:00

Mononoke

Mononoke is a next-generation server for the Mercurial source control system, meant to scale up to accepting thousands of commits every hour across millions of files. It is primarily written in the Rust programming language.

Caveat Emptor

Mononoke is still in early stages of development. We are making it available now because we plan to start making references to it from our other open source projects.

The version that we provide on GitHub does not build yet.

This is because the code is exported verbatim from an internal repository at Facebook, and not all of the scaffolding from our internal repository can be easily extracted. The key areas where we need to shore things up are:

  • Full support for a standard cargo build.
  • Open source replacements for Facebook-internal services (blob store, logging etc).

The current goal is to get Mononoke working on Linux. Other Unix-like OSes may be supported in the future