sapling/eden/scm/lib
Jun Wu 99511f8743 dag: benchmark dag_ops on different IdDagStores
Summary:
Change dag_ops benchmarks to use different IdDagStores. An example run shows:

  benchmarking dag::iddagstore::indexedlog_store::IndexedLogStore
  building segments (old)                           856.803 ms
  building segments (new)                           127.831 ms
  ancestors                                          54.288 ms
  children (spans)                                  619.966 ms
  children (1 id)                                    12.596 ms
  common_ancestors (spans)                            3.050 s
  descendants (small subset)                         35.652 ms
  gca_one (2 ids)                                   164.296 ms
  gca_one (spans)                                     3.132 s
  gca_all (2 ids)                                   270.542 ms
  gca_all (spans)                                     2.817 s
  heads                                             247.504 ms
  heads_ancestors                                    40.106 ms
  is_ancestor                                       108.719 ms
  parents                                           243.317 ms
  parent_ids                                         10.752 ms
  range (2 ids)                                       7.370 ms
  range (spans)                                      23.933 ms
  roots                                             620.150 ms

  benchmarking dag::iddagstore::in_process_store::InProcessStore
  building segments (old)                           790.429 ms
  building segments (new)                            55.007 ms
  ancestors                                           8.618 ms
  children (spans)                                  196.562 ms
  children (1 id)                                     2.488 ms
  common_ancestors (spans)                          545.344 ms
  descendants (small subset)                          8.093 ms
  gca_one (2 ids)                                    24.569 ms
  gca_one (spans)                                   529.080 ms
  gca_all (2 ids)                                    38.462 ms
  gca_all (spans)                                   540.486 ms
  heads                                             103.930 ms
  heads_ancestors                                     6.763 ms
  is_ancestor                                        16.208 ms
  parents                                           103.889 ms
  parent_ids                                          0.822 ms
  range (2 ids)                                       1.748 ms
  range (spans)                                       6.157 ms
  roots                                             197.924 ms

  benchmarking dag::iddagstore::bytes_store::BytesStore
  building segments (old)                           724.467 ms
  building segments (new)                            90.207 ms
  ancestors                                          23.812 ms
  children (spans)                                  348.237 ms
  children (1 id)                                     4.609 ms
  common_ancestors (spans)                            1.315 s
  descendants (small subset)                         20.819 ms
  gca_one (2 ids)                                    72.423 ms
  gca_one (spans)                                     1.346 s
  gca_all (2 ids)                                   116.025 ms
  gca_all (spans)                                     1.470 s
  heads                                             155.667 ms
  heads_ancestors                                    19.486 ms
  is_ancestor                                        51.529 ms
  parents                                           157.285 ms
  parent_ids                                          5.427 ms
  range (2 ids)                                       4.448 ms
  range (spans)                                      13.874 ms
  roots                                             365.568 ms

Overall, InProcessStore > BytesStore > IndexedLogStore. The InProcessStore
uses `Vec<BTreeMap<Id, StoreId>>` for the level-head index, which is more
efficient on the "Level" lookup (Vec), and more cache efficient (BTree).
BytesStore outperforms IndexedLogStore because it does not need to verify
checksum on every read access - the checksum was verified at store creation
(IdDag::from_bytes).

Note: The `BytesStore` is something optimized for serialization, and hasn't been sent.

Reviewed By: sfilipco

Differential Revision: D23438174

fbshipit-source-id: 6e5f15188e3b935659ccde25fac573e9b963b78f
2020-09-02 18:54:12 -07:00
..
async-runtime async-runtime: crate for async job scheduling from blocking threads 2020-08-26 00:57:32 -07:00
auth auth: allow specifying a CA certificate bundle 2020-07-16 19:48:36 -07:00
backingstore configs: make backingstore load hg configs through the approved path 2020-08-16 16:56:00 -07:00
blackbox blackbox: address clippy warnings 2020-03-18 10:16:39 -07:00
bookmarkstore types: address clippy warnings 2020-03-16 14:58:21 -07:00
cdatapack cmake: always build support for the EdenSCM backing store 2019-11-22 13:00:07 -08:00
clib cmake: always build support for the EdenSCM backing store 2019-11-22 13:00:07 -08:00
clidispatch dispatch: add --trace flag 2020-08-27 18:14:28 -07:00
cliparser debug: introduce error classification and metadata propagation 2020-07-15 10:03:10 -07:00
commitcloudsubscriber silence some Rust warnings 2020-08-05 00:16:33 -07:00
commitstore/bench-serialize cargo: bump serde_cbor to 0.11 2020-02-21 14:08:43 -08:00
configparser configparser: make code compile without "fb" feature 2020-08-27 18:28:46 -07:00
cpython-ext taggederror: Introduce taggederror-util for more ergonomic error tagging for eden error types. 2020-08-06 19:37:25 -07:00
dag dag: benchmark dag_ops on different IdDagStores 2020-09-02 18:54:12 -07:00
dev-logger dev-logger: a simple library to enable env_logger for testing 2020-03-10 14:16:38 -07:00
drawdag Use types from the oxidized_by_ref crate instead of the oxidized crate for to_oxidized 2020-05-28 22:19:13 -07:00
edenapi tools: add location-to-hash command to read_res 2020-09-02 17:20:43 -07:00
edenfs-client build: rename third-party rust fbthrift crate 2020-08-26 12:26:21 -07:00
encoding Move fb-mercurial sources into an eden/scm subdirectory. 2019-11-13 16:04:48 -08:00
fsinfo fsinfo: fix cargo test 2020-08-14 13:38:41 -07:00
fsyncglob hgcommands: add debugfsync 2020-08-27 18:26:03 -07:00
hgcommands hgcommands: make commands! macro define modules 2020-08-27 19:02:27 -07:00
hgcommits hgcommits: add flush_commit_data API 2020-08-31 11:57:53 -07:00
hgtime hgtime: address clippy warnings 2020-03-16 14:58:22 -07:00
http_client http_client: disable ssl revocation checking on Windows 2020-08-13 23:17:28 -07:00
indexedlog indexedlog: remove chown feature 2020-04-27 15:47:59 -07:00
linelog Move fb-mercurial sources into an eden/scm subdirectory. 2019-11-13 16:04:48 -08:00
lz4-pyframe lz4-pyframe: use concrete error types 2020-08-06 12:31:56 -07:00
manifest manifest: add testutil in core package 2020-01-14 11:49:52 -08:00
manifest-tree Deprecate rust-crypto in manifest-tree 2020-03-23 11:15:58 -07:00
metalog metalog: implement compaction api 2020-06-26 09:58:58 -07:00
mincode Move fb-mercurial sources into an eden/scm subdirectory. 2019-11-13 16:04:48 -08:00
minibench mononoke/mercurial: make revlog crate OSS buildable 2020-04-08 09:49:11 -07:00
minibytes minibytes: address clippy warnings 2020-03-16 14:58:22 -07:00
mpatch Move fb-mercurial sources into an eden/scm subdirectory. 2019-11-13 16:04:48 -08:00
mpatch-sys use absolute includes for the native cext modules 2020-02-19 13:05:06 -08:00
mutationstore mutationstore: add a native path to calculate 'obsolete()' 2020-08-21 13:00:45 -07:00
nodemap indexedlog: replace Log's dir with a new GenericPath type 2020-01-17 03:57:59 -08:00
pathmatcher Replace whitelist/blacklist term 2020-06-15 15:01:19 -07:00
procinfo procinfo: address compiler warning 2020-03-04 09:49:14 -08:00
radixbuf radixbuf: use concrete error types 2020-08-06 12:31:57 -07:00
renderdag renderdag: move to dag 2020-08-21 13:00:45 -07:00
revisionstore revisionstore: use async-runtime crate for lfs 2020-09-02 10:01:08 -07:00
revlogindex nameset: make Id{Static,Lazy}Set require Dag on construction 2020-08-26 15:32:22 -07:00
spawn-ext spawn-ext: extend Command::spawn to avoid inheriting fds 2020-08-31 17:34:48 -07:00
taggederror taggederror: Introduce taggederror-util for more ergonomic error tagging for eden error types. 2020-08-06 19:37:25 -07:00
taggederror-util taggederror: Introduce taggederror-util for more ergonomic error tagging for eden error types. 2020-08-06 19:37:25 -07:00
third-party build: rename third-party rust fbthrift crate 2020-08-26 12:26:21 -07:00
thrift-types Remove crate_root from Rust dependency info 2020-08-31 14:43:24 -07:00
tracing-collector tracing-collector: limit maximum count of spans 2020-08-27 18:14:29 -07:00
treestate dirstate: block addition of paths containing "." and ".." 2020-08-28 09:42:25 -07:00
types edenapi: Split DataEntry into FileEntry and TreeEntry 2020-08-13 10:01:40 -07:00
util bgprogress: Stdio is only used on Unix 2020-08-21 13:00:45 -07:00
version configparser: fix non-fb dependencies (#45) 2020-08-26 16:31:00 -07:00
vfs fsinfo: recognize EdenFS mounts on Windows 2020-08-12 15:47:49 -07:00
vlqencoding mutationstore: move MutationEntry type to types crate 2020-04-23 08:58:10 -07:00
workingcopy filesystem: add treestate walking logic 2020-04-24 13:58:53 -07:00
xdiff xdiff: renamed third-party xdiff functions 2020-07-09 01:20:32 -07:00
xdiff-sys Update libra to latest revision 2020-07-16 21:10:44 -07:00
zstdelta Move fb-mercurial sources into an eden/scm subdirectory. 2019-11-13 16:04:48 -08:00
zstore zstore: add simple caching 2020-05-01 14:24:52 -07:00
CMakeLists.txt cmake: do not compile configparser 2020-08-06 09:00:20 -07:00
README.md Move fb-mercurial sources into an eden/scm subdirectory. 2019-11-13 16:04:48 -08:00

lib

Any native code (C/C++/Rust) that Mercurial (either core or extensions) depends on should go here. Python code, or native code that depends on Python code (e.g. #include <Python.h> or use cpython) is disallowed.

As we start to convert more of Mercurial into Rust, and write new paths entrirely in native code, we'll want to limit our dependency on Python, which is why this barrier exists.

See also hgext/extlib/README.md, mercurial/cext/README.mb.

How do I choose between lib and extlib (and cext)?

If your code is native and doesn't depend on Python (awesome!), it goes here.

Otherwise, put it in hgext/extlib (if it's only used by extensions) or mercurial/cext (if it's used by extensions or core).