Summary: Sometimes the graph is too large. Provide a way to slice it.
Reviewed By: sfilipco
Differential Revision: D19511575
fbshipit-source-id: 504317d6894764043b23ea49dcf09c8cdea96961
Summary:
As we plan to test the dag crate with some other DAG implementation,
add a convenient structure that setups both DAG implementations.
Reviewed By: sfilipco
Differential Revision: D19503371
fbshipit-source-id: 3e9933ad37301bfac36eb1af6d82b4298af778b6
Summary:
The ported algorithm will work as a comparison to verify dag's gca
implementation.
Reviewed By: sfilipco
Differential Revision: D19503373
fbshipit-source-id: f5253db89fbcdc2fd02f3fdaa0796e24338b1fba
Summary:
This is similar to D17581248. It will make the old linear-scan algorithm (which
will be added later) about 5x faster.
Reviewed By: sfilipco
Differential Revision: D19503372
fbshipit-source-id: c65d7217e7b144603dadd57f54a5e70f513c8e51
Summary: This allows bindag to be used outside benches.
Reviewed By: sfilipco
Differential Revision: D19503374
fbshipit-source-id: 131061f7d1d28125875a86afc330dbb9634249cf
Summary:
We've seen a case where a datapack contains a circular delta chain, causing
Mercurial to fall into a infinite loop when trying to read it. Let's fail when
the chain is over 1000 entries.
Reviewed By: quark-zju
Differential Revision: D19458453
fbshipit-source-id: bfa503f7807122eca72cf94418abda161dafa41c
Summary: This ensures IdMap and IdDag are guaranteed consistent in the storage layer.
Reviewed By: DurhamG
Differential Revision: D19432658
fbshipit-source-id: 00f1a9b4c747baa1f14d78c31d925682317463b4
Summary: This makes it possible to do extra sanity checks.
Reviewed By: DurhamG
Differential Revision: D19443783
fbshipit-source-id: 254c2537a6aadd25a67c5e48a768187ce65aa686
Summary: This makes the code overall shorter.
Reviewed By: DurhamG
Differential Revision: D19443552
fbshipit-source-id: abd1db227830a88549c7eca1cfd08b67c4914518
Summary:
As reported by JT, EdenFS may hold the file descriptor of mapped pack files too long even when it is deleted by external processes, thus taking more disk spaces.
This diff fixes this problem by making EdenFS periodically rescan the pack files.
Reviewed By: chadaustin
Differential Revision: D19395439
fbshipit-source-id: 4bfd6a7ac13dceb3099d2704d62b3825433aff4b
Summary:
In both cases (clone with or without dirty content), the external key buffers
used by indexes should be re-created, since mem_buf location has changed.
Reviewed By: DurhamG
Differential Revision: D19432657
fbshipit-source-id: fe6f76e7ccfd16ccd2f5c1d89866687a3503603e
Summary: This allows us to build the multi-log structure.
Reviewed By: DurhamG
Differential Revision: D19431786
fbshipit-source-id: e0e09b0d5a73d293a80626924b74ddf2ce6d6fa3
Summary: Migrate to the new API. This is more compatible with the next change.
Reviewed By: DurhamG
Differential Revision: D19431788
fbshipit-source-id: dba1a88fd003d17fc1774b0cec9157d32c5acdb0
Summary:
This allows us to incrementally abstract away parts related to filesystem from
Log. For example, instead of using std::fs to access filesystem directly, use
methods on the GenericPath instead.
Right now the main motivation is to add support for "multi-logs" sharing a single meta
file. To do that, methods reading and writing metas are moved to the
GenericPath type.
This also unifies the open functions. Now OpenOptions::create_in_memory can be
private. Empty in-memory Logs can be opened via `open(())`.
Reviewed By: DurhamG
Differential Revision: D19431784
fbshipit-source-id: dbdf94f60261e09f131c6fdd9fe3b99242a28af5
Summary: This makes an upcoming change easier.
Reviewed By: DurhamG
Differential Revision: D19432656
fbshipit-source-id: 65adc883c3c3937aa7022196b83c75715b820721
Summary: The goal is to split log.rs to smaller modules so it's easier to reason about.
Reviewed By: DurhamG
Differential Revision: D19431787
fbshipit-source-id: 23b8346c461da322e7a2240b8224e6194867aaee
Summary:
The goal is to split log.rs to smaller modules so it's easier to reason about.
OpenOptions has many dependencies. Move the most complicated one (repair) to
another standalone module.
Reviewed By: DurhamG
Differential Revision: D19431783
fbshipit-source-id: 28c3af7cd6e19588bfe9c395e2648244faf3179d
Summary: The goal is to split log.rs to smaller modules so it's easier to reason about.
Reviewed By: DurhamG
Differential Revision: D19431785
fbshipit-source-id: b9a598b3018267ff6e43ef57dc807cb4e880467c
Summary:
In order to avoid pathological linkrevfixup after pushing a change to a
pushrebase repo, we need to be able to fetch history data that is already
present locally from the server. Since the Rust stores will always check
whether the data is present locally before fetching it, we would not be
fetching anything, causing the pathological linkrevfixup to kick in.
By allowing the stores to be built without a local component, the prefetch
function will not find the local history, and will thus be able to fetch it
properly.
Reviewed By: DurhamG
Differential Revision: D19412619
fbshipit-source-id: 421c59c63634ead7f98e6ba89da0532067f7b412
Summary:
In the cpython bindings, the Rust String can take both PyBytes and PyUnicode
strings, which is perfect for Python3 compatibility as string literals are
PyBytes in Python2 and PyUnicode in Python3. The return values are kept as
Bytes for now as changing this is a much larger change in itself.
Other approaches tried:
- Using PyUnicode as input/output: an extremely large codemod had to be done,
with very little benefits
- Using String as output: since we do have some configs that are unicode, on
the Python side, the output might either be bytestrings or unicode, leading
to weird bugs.
Reviewed By: DurhamG
Differential Revision: D18650466
fbshipit-source-id: aebdf30590dcae40b7df2787e5ece88e2ec9395c
Summary:
Mercurial assumes some specific ordering, let's comment about this to make sure
we don't re-order and introduce subtle bugs.
Reviewed By: quark-zju
Differential Revision: D19394352
fbshipit-source-id: 0f9e02d2c6addf040311a54b8161b06bbeaa6be9
Summary:
The error message and the exception type are slightly different between the
Rust and Python ContentStore. For now, let's just fix this up manually.
Reviewed By: quark-zju
Differential Revision: D19394350
fbshipit-source-id: e432094a9dfcf605568a1890c0303b733e98d203
Summary: The rust telemetry wrapper and the python code were not in sync in how they presented errors to the user on failure. This makes them identical in message and error code (however the "abort" from python is red in color).
Reviewed By: quark-zju
Differential Revision: D19313848
fbshipit-source-id: d5b95e8d5130f20fc85c6664933ee18702f7cc5e
Summary:
Previously, NameDag only supports writing directly to disk. That is not easy
for hg to use, since hg knows commits before bookmark names during pull, while
the API requires both commits and bookmarks (to decide which commits belong to
the "master" group) at the same time.
The old API also does not match other storage layers like indexedlog and
metalog, where writes are buffered until explicitly flushed.
This diff adds the "write-in-memory" (named "add_heads"), and "flush" APIs to
make NameDag easier to use. Under the hood, it just copies the "added" portion
of the DAG, and re-use the old API ("build", renamed to "add_heads_and_flush")
to do the job.
Note: The buffered changes pattern cannot be used in IdDag, since Ids might
have to be re-assigned on flush. But NameDag does not expose the raw Ids in its
"normal" interface, so it's fine to reassign Ids on flush.
Reviewed By: markbt
Differential Revision: D19405474
fbshipit-source-id: 75e5e5815c78c3577a0138f48185f6c4b5a80891
Summary:
This is more consistent with the name "IdDag", because both "Name" and "Id" are
noun.
Reviewed By: singhsrb
Differential Revision: D19405473
fbshipit-source-id: a3b7546dd0ddcae5d9ed562838bd6be10a47063c
Summary: This is more consistent with the name "NamedDag".
Reviewed By: singhsrb
Differential Revision: D19405472
fbshipit-source-id: f7023307acaf96bf77c9fa9704dcaf6fc59b56f2
Summary:
The git repo has a lot of merges. So it's interesting as a testing graph.
The file was generated by:
git clone https://github.com/git/git --bare
hg --config extensions.hggit= clone git.git git-hg
hg --cwd git-hg debugbindag -r 'all()' -o git.bindag
Reviewed By: DurhamG
Differential Revision: D19411825
fbshipit-source-id: 0fec8a16786dc8e6790986bca5c62a76276aa942
Summary:
This changes the pattern commonly seen in smartlog:
╷ o e7f5f529 ...
╭─╯
│ o a1b2773d ...
╭─╯
o ecbb4eaa
╷
to:
╷ o e7f5f529 ...
├─╯
│ o a1b2773da ...
├─╯
o ecbb4eaa
╷
The change only applies to commits with a single parent. Vertical lines from
merge commits stay disconnected intentionally. For example:
│ │ o I
│ │ │
│ o │ H
╭─┼─┬─┬─╮
│ │ │ │ o G
│ │ │ │ │
This makes it more obvious that `H` has 5 parents while `I` only has 1 parent -
`I` does not "borrow" `H`'s parents. This problem does not exist if `H` only
has 1 parent.
Reviewed By: quark-zju
Differential Revision: D19407687
fbshipit-source-id: 1046c8e2309f50e3f1620ed21f1b10573759a5f8
Summary:
Moving `store_element` and `get_hgid` from testutil.rs to lib.rs in order to
get rid of some buck build warnings. They are only used in lib.rs.
Reviewed By: markbt
Differential Revision: D19350343
fbshipit-source-id: 081783d753425daaae6fadc38589da5e4e8b745d
Summary:
It makes sense for manifest-tree to expose a testutil version for building a
TreeManifest. No reason for the testutil function to be limited to the crate.
The only issue is that `make_tree` is to generic so we rename the function to
`make_tree_manifest`.
Reviewed By: markbt
Differential Revision: D19350351
fbshipit-source-id: 06fe04d99bf820d3e81417f5a5514aa4b0d282e6
Summary:
Types that are defined by the manifest core crate should have test utilities
defined in the same crate.
This is motivated by various warning in the buck build.
Reviewed By: markbt
Differential Revision: D19350350
fbshipit-source-id: a6a7c3fb54b465aa09a28ff8b70b49a355b328fc
Summary:
Previously, if the rust walker encountered any issues with individual
files it would stop and throw an exception or in some cases panic. To replace
the python walker we need to handle errors more gracefully, so let's add logic
for reporting which files had issues and what those issues were.
This also contains a minor one-line fix to prevent the walker from traversing
directories that contain a '.hg' directory.
Reviewed By: quark-zju
Differential Revision: D19376540
fbshipit-source-id: ede40f2b31aa1e856c564dd9956b9f593f21cf27
Summary:
This simplifies the error handling and makes it more compatible with things that
might capture the output, including the Python testing framework.
Reviewed By: DurhamG
Differential Revision: D19325642
fbshipit-source-id: 53de8b9a8294219e2b8e62831dce236841bd4cbb
Summary: Make `hg status` use the EdenFS Thrift path, similar to the telemetry wrapper.
Reviewed By: DurhamG
Differential Revision: D19325641
fbshipit-source-id: 14777a252d7cb433316511a2a1f1a6649e9cb020
Summary:
Make it possible to define reusable flags like walkopts, templateopts that can
be shared among commands.
Reviewed By: DurhamG
Differential Revision: D19325640
fbshipit-source-id: 89a9ad0e59672b1f8cfa22daaacd3c75d8cff61a
Summary:
Add a crate aiming for talking to EdenFS via Thrift.
This is mostly ported from telemetry. There are lots of things that can be
cleaned up like moving the command portion to hgcommands, or using a better
abstracted rendering layer for outputting, etc. But for now, I just ported
the code as-is.
Reviewed By: DurhamG
Differential Revision: D19325639
fbshipit-source-id: ac70315ed166024857a90aab070e4d521f3e3b58
Summary:
The main motivation is to replace (super slow) Python Thrift serialization.
It also makes it possible to port the native "status" command from telemetry
wrapper to the main executable.
The crate is made to be built with either buck or cargo in fbcode environment:
- Cargo build uses `build.rs` to re-generate (checked-in) Thrift code.
- Buck build uses TARGETS to build Thrift code on the fly.
Reviewed By: DurhamG
Differential Revision: D19325638
fbshipit-source-id: 1d3287d8d6c674065500929827fa13ba9d448708
Summary:
Enhance the tests to test whether the width of the graph matches the width that
was given by the `width` method before `next_row` was called.
The width we are interested in is the number of cells the string would occupy,
not the string length, so use the `unicode-width` crate to determine this.
Reviewed By: quark-zju
Differential Revision: D19282571
fbshipit-source-id: d9852c4c9e0f76c78db047f0da5dd34723a62a2a
Summary: Switch from `:` to `.` for the ancestor lines. I think they look better.
Reviewed By: quark-zju
Differential Revision: D19371721
fbshipit-source-id: e18f1a62e23620a82007e2c377607a0a61623830
Summary:
Box-drawing characters with curves aren't reliably renderable on Windows. Add
a "square" glyph set that uses right-angle characters. These characters are
available in cp437 and cp850, so should be available on most Windows systems.
For completeness, add a "dec" glyph set that uses DEC line drawing characters,
for use in old terminals like xterm.
Reviewed By: quark-zju
Differential Revision: D19371722
fbshipit-source-id: 35887243cceab66c702e2b5278b572f77946805f
Summary:
Using RepoPath::arbitrary() has a high chance or producing a lot of different
top level directories. The helper method added by this change,
`generate_repo_paths` simulates directories having various files and several
directories.
Reviewed By: quark-zju
Differential Revision: D19321366
fbshipit-source-id: 5c1aec78b6157f3cbea3d0673b29b3a676de88c0
Summary:
The TestStore can be leveraged by methods outside the manifest-tree crate.
Anything that wants to instantiate a manifest for test purposes could
leverage it.
Reviewed By: quark-zju
Differential Revision: D19204894
fbshipit-source-id: 4ac42d09855c70f829feefc6c71dcdbf7211cae3
Summary:
If the user specifies a timezone-less date, ex. `2000-7-1`, it's incorrect
to use the same offset of `Local::now()`, because of daylight savings.
Fix it by going through proper type conversion.
Reviewed By: DurhamG
Differential Revision: D19179800
fbshipit-source-id: 6fd433757e7f6e4baf4625944d4061548a436f11
Summary:
Previously, between 100 to 500 runs, the test is likely to fail:
---- log::tests::test_repair_and_delete_content stdout ----
thread 'log::tests::test_repair_and_delete_content' panicked at 'assertion failed: `(left == right)`
left: `"Verified first 141 entries, 999 of 1400012 bytes in log\nReset log size to 999\nIndex \"c\" is incompatible with (truncated) log\nRebuilt index \"c\""`,
right: `"Rebuilt metadata\nVerified first 141 entries, 999 of 1400012 bytes in log\nReset log size to 999\nRebuilt index \"c\""`', src/log.rs:3414:9
The "Rebuilt metadata" part is missing.
This is because after D17742003, the end of `meta` is the `epoch`, which is a
random number. The test is rewriting the end of `meta` to corrupt it. That
has a chance to not corrupt the file. Change the test to corrupt the fixed
header of meta so it can always corrupt the file and therefore stabilize
the test.
Reviewed By: xavierd
Differential Revision: D19325087
fbshipit-source-id: 53d46c8a6cb9771582f8f37e27f185d303fde0ce
Summary:
Per discussion, we decided to use "VertexName" as the struct name for things
like commit hashes, or the string names in tests (or the Mozilla DAG in tests).
Therefore, introduce a dedicated VertexName type and repalce all callsites to
use it.
`bytes::Bytes` is used so copying the `VertexName` is somewhat considered cheap.
This adds some overhead copying slices (and `Bytes` has some overhead). It
regresses the "building segments" benchmark from 673ms to 773ms, which seems
okay given the cleaner interface.
Reviewed By: markbt
Differential Revision: D19154905
fbshipit-source-id: 4c6d4eca67c11c10ed5f21999ccdc3f1b01695e8
Summary:
Redefine Id::{MIN, MAX} using Group so Id::MAX.group() is a valid group.
Previously Id::MAX has an invalid group.
Reviewed By: sfilipco
Differential Revision: D19154908
fbshipit-source-id: 6108f02da786eb228a7d1b772ca5134c1ebb6f6b
Summary:
Address review feedback on D18670160.
This also fixes the definition of "universal" a bit - the new code no longer
assumes there are only one head in the master group.
Reviewed By: sfilipco
Differential Revision: D19154906
fbshipit-source-id: 4fbf080354318300ce09dc7fba2dcc5f942b6308
Summary:
The check of "is this high level segment necessary or not" should be done
before dropping the last segment. This fixes an issue discovered in D18838992.
Reviewed By: sfilipco
Differential Revision: D19154904
fbshipit-source-id: fb4c83c39d66215bae168ad98e5cf78de91cc5a3
Summary:
NamedDag is the high-level interface that is more interesting to test.
High-level segments are changed subtly because NamedDag syncs them to disk
first.
Reviewed By: sfilipco
Differential Revision: D18838992
fbshipit-source-id: c6a557e0a44a1d24320ea4a9e4283262f6f30f67
Summary: Those warnings are not seen on POSIX build.
Reviewed By: sfilipco
Differential Revision: D19320441
fbshipit-source-id: feddf728eb9627834559b87d83e20f0afd9080c8
Summary:
Move the test fixtures into a common module, so that they don't need to be
repeated in each test. Since each fixtures is now a struct, this also makes it
clearer what each of the items are.
Reviewed By: quark-zju
Differential Revision: D19288290
fbshipit-source-id: 394805c652592177f11ccb096b8e5e95361456e4
Summary:
Generalize construction of output renderers (renderers that render to `String`)
to avoid duplication of options. At the moment there is only one, but later we
may add new options.
This also allows us to construct output renderers from any renderer that
renders to `GraphRow`, which means we can create adapters that modify or
re-order the rows of the graph.
Reviewed By: quark-zju
Differential Revision: D19286350
fbshipit-source-id: a5649ca2f48e263ee24584339179655fb612d3d1
Summary:
A new implementation for rendering DAGs.
This new crate implements a generic DAG renderer, that can convert a
topologically sorted sequence of DAG nodes into a sequence of strings suitable
for rendering to a terminal.
The new renderer differs from the old renderer in a few important ways:
* It prioritizes keeping commits linear, and will allow gaps to form if
that will allow the history of the commits to be kept in a straight
line. This makes it easier to track long parallel histories.
* It supports octopus merges (nodes with more than two parents). Even
though Mercurial doesn't support octopus merge commits, summary DAGs
with omitted nodes can still end up with logical octopus merges.
* It supports reservation of columns for specific nodes. This can be
used to support smartlog-style indentation of draft stacks without
needing to hack around it by creating fake nodes.
* It separates out forming the graph from generating the lines. This
allows multiple back-ends for generating different styles of graph.
There are three back-ends implemented:
`AsciiRenderer` renders similar to the old graph renderer, using ASCII
characters. For example:
```
o F
|
| o E
| |
| | o D
.-----'
o | | C
+---'
o | B
:/
o A
```
`AsciiLargeRenderer` uses larger ASCII blocks to give a clearer picture of
complex graphs. For example:
```
o F
|
|
| o E
| |
| |
| | o D
______/
/ | |
o | | C
| ___/
|/ |
o | B
: /
:/
o A
```
`BoxDrawingRenderer` uses Unicode box drawing characters to give a more
continuous rendering of the graph, however requires support for these
characters in the terminal font. For example:
```
o F
│
│ o E
│ │
│ │ o D
╭─────╯
o │ │ C
├───╯
o │ B
├─╯
o A
│
~
```
Reviewed By: quark-zju
Differential Revision: D19272579
fbshipit-source-id: bb6fa4685c965544cc3b6b9261df3a3ec161b41f
Summary:
The existing tests rely on this feature.
This requires replacing all `Local::` usage with `HgTime::now()`.
Reviewed By: sfilipco
Differential Revision: D18946336
fbshipit-source-id: 1a92183cc96f2fcb013b68a32e78f09f28a49452
Summary:
For example, `2018 to 2019` should be parsed as `2018-1-1 to 2019-12-31 23:59:59`,
not `2018-1-1 to 2019-1-1`.
Fix it to match the desired behavior.
Reviewed By: sfilipco
Differential Revision: D18946331
fbshipit-source-id: 640bbe49352d1f01855183ea9927bb0009646243
Summary:
The Python `datatime` stdlib has limitations:
- `datetime.datetime` can only express year 1 to 9999.
- `strptime` requires year >= 1900.
Limit the `HgTime` to 1900 ..= 9999 range to be consistent.
Note: Mercurial has an `i32` range limit, which seems to problematic
since `i32::MAX` is Jan 19, 2038. So I kept using `i64`.
This addressed some XXX comments about not returning errors.
The code change is mostly because added error handling.
Reviewed By: sfilipco
Differential Revision: D18946333
fbshipit-source-id: 0e4756457b0f13451dc5008ef19d4670a7aaa7fb
Summary:
It turns out the interaction between dag and idmap can be complex, especially
when a non-master id gets re-assigned a master id.
A high-level structure is helpful here to handle the corner cases correctly.
This diff adds basic definitions, and one of the most complex function - build.
The build function is ported from pydag.
Reviewed By: sfilipco
Differential Revision: D18838994
fbshipit-source-id: a2f6db3b188d5fd04d8e1d6df40ba985e33281f6
Summary: First step to implement segment and idmap rewrites for non-master ids.
Reviewed By: sfilipco
Differential Revision: D18838996
fbshipit-source-id: 0380c0193d9078a2a2d7fde9c5f565e8dbc8e713
Summary:
The index can delete all keys matching a prefix more efficiently than deleting
them one by one. Expose this feature.
The `dag` crate will use this feature to delete all "non-master" segments and
ids efficiently.
Reviewed By: sfilipco
Differential Revision: D18825296
fbshipit-source-id: b8531695609238a16913254af61004170f12954e
Summary: This makes it possible to define an index function that can delete data.
Reviewed By: sfilipco
Differential Revision: D18825299
fbshipit-source-id: 4eb03a001683092d99b439c8efb8e7909f543c70
Summary:
This makes it possible to actually delete keys from the index, which can be
handy for certain use-cases. For example, `dag` wants to rewrite non-master
ids and segments in certain cases.
Reviewed By: sfilipco
Differential Revision: D18825298
fbshipit-source-id: 3b69ca5fc763ba0c57bd16a28a349f2f829a0759
Summary:
This just moves parameters to a structure without changing the actual logic.
This is to make it possible to add a "remove" API to index.
Some variables are prefixed with `old_` or `new_` to make the code more
consistent.
Reviewed By: sfilipco
Differential Revision: D18825297
fbshipit-source-id: 27bd8438e66125c906ebf2044d6ff978b99c5986
Summary:
Make cpython-ext free from business logic. This adds a bit overhead.
But it should be fine given the error cases are considered rare.
Reviewed By: markbt
Differential Revision: D19186692
fbshipit-source-id: daaffc1369a36c781d1badea822bf62a144eb54e
Summary: We no longer use the `failure` crate. Rename related methods and traits.
Reviewed By: markbt
Differential Revision: D19186694
fbshipit-source-id: 1c83a90ee12db431b7d8e09a9c2cf1d43a4c0f93
Summary:
We no longer use the `failure` crate. Rename the module.
`cpython-failure` is dropped since it was merged into `cpython-ext`.
Reviewed By: markbt
Differential Revision: D19186693
fbshipit-source-id: 410d1491bcadd8d3272c7e2df08ecbe66fc0fef2
Summary:
The `map_pyerr<PE>` API is kind of hard to use correctly since the
code that calls `map_pyerr` does not really know the best suitable error type.
The existing code also "encourage"s the use of stdlib error types like
RuntimeError, ValueError, etc since the code is shorter, which is not great.
Error types from `mercurial.error` are better choices.
With previous diffs in this stack, we now decide the Python error type based
on the Rust error type. It's no longer necessary to specify an error type
with `map_pyerr<PE>`. So let's just drop the type parameter.
A `RustError` was added for error types that `cpython-ext` does not know how
to convert. We should avoid leaking it to the Python land by just implementing
how to convert `anyhow::Error` to `PyErr` in `cpython-ext`.
Reviewed By: markbt
Differential Revision: D19169527
fbshipit-source-id: f4563c36174cd51201b526bbc92a3f1c8a3da864
Summary:
Convert indexedlog error and IOError to dedicated Python types so the Python
world can match them meaningfully by type and handle them accordingly.
Reviewed By: markbt
Differential Revision: D19169210
fbshipit-source-id: 1e7bfdf7cbaf917098efdcd63d600483a9427d33
Summary:
When there are many processes (ex. import helpers). Writing to blackbox can be
slow due to flock contention.
Change blackbox "flush" decision from timestamp based to buffer-size based to
mitigate the issue.
Reviewed By: DurhamG
Differential Revision: D19198470
fbshipit-source-id: 71f87e5e3b45ef4db8efda22ca89fd8ce1fae7fa
Summary:
The current implementation iterates through all entries, which will include
deleted entries. Rewrite it to use index lookups so it does not print
deleted entries.
Reviewed By: sfilipco
Differential Revision: D18838993
fbshipit-source-id: 58d3f2da27cd3e91bc10cbb04bcdca3b7ff07dbb
Summary:
Address review comment in D18640899. I went a bit further that makes COUNT,
ALL, and BITS consistent.
Reviewed By: sfilipco
Differential Revision: D18820721
fbshipit-source-id: e354a3aecf0d8cbe66c43d56198933cdc6420241
Summary:
Address review comment in D18640898.
Id::min_value and Id::max_value become unused and are removed.
Reviewed By: sfilipco
Differential Revision: D18820722
fbshipit-source-id: d9682bf5918307166571fa35aa15cc9eee1a7140
Summary:
Address review comment in D18640899. This makes the word `id` universally
refer to a same concept within the crate.
Reviewed By: sfilipco
Differential Revision: D18820723
fbshipit-source-id: 6803192db7e1304a72100568f8f29b90f25c7779
Summary: There is no need to insert duplicated entries to IdMap.
Reviewed By: sfilipco
Differential Revision: D18670158
fbshipit-source-id: 812ea421c9f36cd12b4aa2be5b2560abec627a52
Summary: This emulates what IdMap the server will send to the client at initial clone.
Reviewed By: sfilipco
Differential Revision: D18670160
fbshipit-source-id: 496142c1e149df9accf816e3d0a1b8316ff64c67
Summary: This makes it easier to check what's stored in IdMap.
Reviewed By: sfilipco
Differential Revision: D18670159
fbshipit-source-id: 91387d01da7d52a0d349e87ffa0aa8983bbe2213
Summary:
This diff implements the following part in a local-only environment:
Id -> Slice: Id -> Request --> Response -> Slice
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Slice -> Id: Slice -> Request --> Response -> Id
^^^^^^^^^^^^^^^^^^^^^^^^^^
Reviewed By: sfilipco
Differential Revision: D18664210
fbshipit-source-id: 740b9cdfe533e7bcc9bbbf2a53c8bcc5bc3648bf
Summary:
This diff implements the following part in a local-only environment:
Id -> Slice: Id -> Request --> Response -> Slice
^^^^^^^^^^^^^
Slice -> Id: Slice -> Request --> Response -> Id
^^^^^^^^^^^^^^^^
Reviewed By: sfilipco
Differential Revision: D18664205
fbshipit-source-id: 47a35c6f95a58ab29c014ac3c6799f16e6114da3
Summary:
In the future the client will have a sparse IdMap, and needs the server (who
has a complete IdMap)'s help to convert between Id and Slice (commit hash)
for commits in the MASTER group.
This diff defines basic abstractions for communications between a sparse IdMap
and a complete IdMap. The basic flows are:
To convert Id to Slice: Id -> Request --> Response -> Slice
To convert Slice to Id: Slice -> Request --> Response -> Id
(`->`: client-side; `-->`: server-side)
Reviewed By: sfilipco
Differential Revision: D18664206
fbshipit-source-id: 93a6b671be30ab1247c71d814817a2ea06b6bc66
Summary: This diff makes EdenFS to be able to use EdenApi to import blobs and trees when requested objects are not present in local hgcache.
Reviewed By: chadaustin
Differential Revision: D18605547
fbshipit-source-id: 4acd2e07cfd9de6b6775ded30ea22a4478b9f1e4
Summary: This will allow us to use EdenApi in EdenFS to fetch trees and blobs.
Reviewed By: xavierd
Differential Revision: D18622844
fbshipit-source-id: 59a9091e9f2fdbcae078da2fb24ee9c0dd18505b
Summary:
Add an option `experimental:use-edenapi` to `EdenConfig`.
See the next diff for usage.
Reviewed By: chadaustin
Differential Revision: D18605549
fbshipit-source-id: 2786c21bb38a76229078662cc5c1ddf906d1be4a
Summary: Ideas for improvements. Put them in the code for future consideration.
Reviewed By: quark-zju
Differential Revision: D18870140
fbshipit-source-id: a2a84d94d72303adc64614b06e36588ff7778aab
Summary:
The list functionality is required by EdenFs. We want this functionality
to be well supported by the Manifest.
Reviewed By: quark-zju
Differential Revision: D18870143
fbshipit-source-id: 1ebaa713ff521226e6ace22cbd35cc841d967298
Summary:
Consistency in naming. The general idea is to have Metadata types that don't
contain paths. Then we will have File, Directory and eventually FsNode that
will contain paths.
Reviewed By: quark-zju
Differential Revision: D18870141
fbshipit-source-id: a1f09add7f1c3dd4fa0348693cd3ce2fd5767fa7
Summary:
This rename is going to make it easier to import and use outside of
manifest specific crates.
Reviewed By: quark-zju
Differential Revision: D18870142
fbshipit-source-id: 2f3ea460170308162ee834efc038b2dcedd9e233
Summary:
Provide a way to repair the MetaLog. It's not just repairing 2 indexedlog
structures, but also checking the relationships between them.
Reviewed By: xavierd
Differential Revision: D18737904
fbshipit-source-id: fcf8ae56a1fdbb0561765701d962dfad4a8b5bd4
Summary:
The environment variable `EDENSCM_TRACE_OUTPUT` specifies where to write the
tracing output for the current command. Environment variables are inherited
by subprocesses by default. That is undesirable because another hg command
(triggered by hook, or background maintaince) will rewrite the trace output.
Avoid it by unsetting EDENSCM_TRACE_OUTPUT for subprocesses.
Reviewed By: xavierd
Differential Revision: D18892973
fbshipit-source-id: 575b6c0df2e7a0775172893e4aa72ca33fa4658c
Summary:
The previously behavior silently uses the empty Root upon "id not found" cases,
which hides data corruption issues captured by the next changes.
Reviewed By: xavierd
Differential Revision: D18737906
fbshipit-source-id: 0f2cfe1b05f11e9d2584d96a14afcb1aed80da4d
Summary: Otherwise the repair will fail with "unable to lock directory" errors.
Reviewed By: xavierd
Differential Revision: D18737911
fbshipit-source-id: 2c47b5b9a071e0db2d20fc385ef466aef1e732f6
Summary: This makes it possible for other crates to implement the `Repair` trait.
Reviewed By: xavierd
Differential Revision: D18737909
fbshipit-source-id: 3d098e583b45e84eb4bd01ef45569e9694b63ed6
Summary:
This simplifies the logic a bit by getting a free `repair` method from
indexedlog.
Reviewed By: xavierd
Differential Revision: D18737908
fbshipit-source-id: 4988c1a83b7709b751cd1899c5663acc0c42e313
Summary:
The goal is to make it easier for different structures to implement `repair` by
just providing `OpenOptions`.
Reviewed By: xavierd
Differential Revision: D18737903
fbshipit-source-id: af3c18038c5bb2f91f53ab53926ec097e1d87e6b
Summary: Uses `folly::IOBuf`'s `TakeOwnership` instead of `CopyBuffer` to achieve zero-copy of file content we get from Rust.
Reviewed By: chadaustin
Differential Revision: D18264900
fbshipit-source-id: 45baa70cb771a2bff0b31feaa1b85569625218eb
Summary:
This isolates the core types in manifest so that it is harder to
create unsound dependencies with specific implementations.
Reviewed By: quark-zju
Differential Revision: D18843133
fbshipit-source-id: 4b866ad84d2e7d0ff2dc4ec6bd65f66548c3fe4a
Summary: Signal more clearly that the iteration in the cursor is depth first.
Reviewed By: quark-zju
Differential Revision: D18843135
fbshipit-source-id: 8ee2c612963eaf6fec3ae48f75ebf16bd177fed4
Summary: Items feels a bit generic as a name.
Reviewed By: quark-zju
Differential Revision: D18843136
fbshipit-source-id: 29a904f3a8a1996760e71bd78610bdbfef4ad65b
Summary: No longer strictly related to files.
Reviewed By: quark-zju
Differential Revision: D18843128
fbshipit-source-id: c7853e08aa5d0256616e777c5bb6dd477b9ebc97
Summary:
`hgid` can be easily derived from the link member variable. Using link is
preferable because link is the owner of the data. It makes the logic more
specific, less bug prone to use the true source of the data.
Reviewed By: quark-zju
Differential Revision: D18843137
fbshipit-source-id: 443f1e5eb4498fa4f2ff9904dba7cc3dc4305add
Summary:
This structure is not public. It is private to the manifest crate so there is
no point to have those modifiers.
Reviewed By: quark-zju
Differential Revision: D18843134
fbshipit-source-id: 0794bbbf578863cabbd85ada87669b791ebcc72d
Summary:
The purpose of this change is to disambiguate from the other Directory struct
that we have in the Manifest. Right now that other is in manifest/lib.rs
Reviewed By: quark-zju
Differential Revision: D18843130
fbshipit-source-id: f25a60f3040cc19abcab99ac5f2cdbff83ccf7ee
Summary:
Directory<'a> which wraps a link is closely linked with the Link abstraction.
It can be seem as a convenience wrapper around Link.
Reviewed By: quark-zju
Differential Revision: D18843129
fbshipit-source-id: 2b677d71101bff3708a65e03bbcddf5074d51387
Summary:
Diff is a core operation for manifests. The Manifest trait should require that
it is implemented.
Reviewed By: quark-zju
Differential Revision: D18843132
fbshipit-source-id: 5e3febab669470e266878d934561dfc323b71c7f
Summary: The conversion from tuple is not used much. I don't like them that much either.
Reviewed By: quark-zju
Differential Revision: D18843131
fbshipit-source-id: fb06b72a279b59ee8af5e43d83a10e39a576730b
Summary:
Querying all the files in a Manifest is core functionality that every Manifest
implementation should implement. The main question is how do we describe that.
We want to return an interator over the files of the manifest. The problem is
that the manifest is a trait and functions on traits can't return traits.
The solution is to Box the results. Not pretty but allows us to move the code
forward.
Reviewed By: quark-zju
Differential Revision: D18822854
fbshipit-source-id: ca44b24816203049f747ad67af48e33370861024
Summary:
The purpose of the rename is to signal that this function may potentially do
a lot of work, even calling and waiting on network requests.
Reviewed By: quark-zju
Differential Revision: D18822855
fbshipit-source-id: 98471944c8e6c7cffa7a2a11f2f733274802b376
Summary:
Removing the contructor for File that takes dependency on
tree implementation details.
Reviewed By: kulshrax
Differential Revision: D18822858
fbshipit-source-id: aed33b146dfdbdb23721a0f6e96977025d71a627
Summary: The types in file.rs being in a separate file confuse people.
Reviewed By: quark-zju
Differential Revision: D18822856
fbshipit-source-id: 29aa10c76bb5ef74f1fc0bf726e6b3cfaf19b428
Summary:
In the future we might have revlog or more complicated storage for maintaining
the IdMap. Define a trait for it.
Reviewed By: sfilipco
Differential Revision: D18664208
fbshipit-source-id: 242611d78852e8d82a52ab52192db87c13b848e7
Summary: This is the reverse of first_ancestor_nth. It will be used later.
Reviewed By: sfilipco
Differential Revision: D18664209
fbshipit-source-id: 8d3706cdc080086ec60da60870fde76c8933f6a8
Summary: This will be used in an upcoming change.
Reviewed By: sfilipco
Differential Revision: D18664207
fbshipit-source-id: 3030bb02e172b95c6dbabda74c318e7f16212f59
Summary:
With the ONLY_HEAD flag, ancestors can be sped up sometimes. Since ancestors
is used widely in other DAG calculations, perf improvements are visible across
the benchmark. The trade-off is to spend a bit more time on building segments
to figure out ONLY_HEAD.
After this diff:
building segments 576.625 ms
ancestors 47.119 ms
children 436.967 ms
common_ancestors (spans) 257.096 ms
descendants (small subset) 367.316 ms
gca_one (2 ids) 146.315 ms
gca_one (spans) 253.059 ms
gca_all (2 ids) 236.075 ms
gca_all (spans) 266.695 ms
heads 220.766 ms
heads_ancestors 36.258 ms
is_ancestor 98.967 ms
parents 219.173 ms
parent_ids 9.282 ms
range (2 ids) 99.859 ms
range (spans) 205.154 ms
roots 429.047 ms
Before this and previous diff:
building segments 542.564 ms
ancestors 316.075 ms
children 428.612 ms
common_ancestors (spans) 452.850 ms
descendants (small subset) 537.483 ms
gca_one (2 ids) 526.112 ms
gca_one (spans) 446.726 ms
gca_all (2 ids) 780.590 ms
gca_all (spans) 490.906 ms
heads 214.005 ms
heads_ancestors 217.558 ms
is_ancestor 256.769 ms
parents 213.170 ms
parent_ids 9.313 ms
range (2 ids) 259.435 ms
range (spans) 358.934 ms
roots 423.005 ms
Reviewed By: sfilipco
Differential Revision: D18642797
fbshipit-source-id: 27f1ee5797a92142b2c19195eb26baef58769553
Summary: The pre-calculated flag can be used for some fast paths.
Reviewed By: sfilipco
Differential Revision: D18642796
fbshipit-source-id: bc16b3517cc29e23478f2027e83e2a2282be0d0a
Summary:
The bindings APIs are out of date.
The latest idea is to write non-master commits to disk, in a different group.
There are no "segments in memory". Therefore:
- Rename `builddisk` to `build`.
- Make sure `build` updates internal states by calling `reload`.
- Drop `buildmem`.
- In case `builddisk` gets called very frequently, reduce its overhead by
doing nothing if all commits are built already.
The constructor is revised to match the latest interface (no
"max segment level" and "set_new_segment_size" needs to be called).
Reviewed By: sfilipco
Differential Revision: D18640897
fbshipit-source-id: 76a5d0db363713dc5b763503c238c1bfa61ea02e
Summary: Make it possible to build non-master segments.
Reviewed By: sfilipco
Differential Revision: D18641507
fbshipit-source-id: 4d0c46cdea240849af37d28219b6023578514323
Summary:
`build_segments_volatile(id)` should build flat segments so it includes `id`.
Right now it can fail that in a corner case. Fix it.
Reviewed By: sfilipco
Differential Revision: D18641509
fbshipit-source-id: ad1de694e4d31c51f29e9bc5800291066bf16dbb
Summary:
The IdMap has assertions about 1:1 mapping between Id and Slice.
Practically, a non-master commit can be merged into a master commit.
That means a commit hash might map to 2 Ids: [master-id, non-master-id].
Allow that to happen (temporarily).
Note: The non 1:1 mapping causes more headache than benefit and
D18838995 restores 1:1 mapping. But it still requires temporary
non-1:1 mapping to work.
Reviewed By: sfilipco
Differential Revision: D18641508
fbshipit-source-id: bc4bc831d5ab5ebfc2588bafd48114ae49d31e0f
Summary: panic on drop is a terrible idea. It can cause SIGILL kill without meaningful messages or meaningful backtrace. Remove it.
Reviewed By: xavierd
Differential Revision: D18640894
fbshipit-source-id: 3ab52ded9875e17e891acd8f9b45d82964615f01
Summary: This makes the code harder to crash in abnormal cases.
Reviewed By: sfilipco
Differential Revision: D18640895
fbshipit-source-id: ee81dd7739739d3d15052bdb7bb2ba80ed48cd0a
Summary:
This allows us to allocate Ids for non-master commits in a separate range
so they wouldn't cause fragmentation for commits in the master group.
Reviewed By: sfilipco
Differential Revision: D18640896
fbshipit-source-id: c19c53342583ded8795849584cf5f31bc86b2818
Summary: Migrate the codebase to use the new Id type instead of an alias of u64.
Reviewed By: sfilipco
Differential Revision: D18640898
fbshipit-source-id: 415db9afeda1f8960e18c69e70f53a9eeb63cbbe
Summary:
Background:
- Tests and Nuclide haven't removed usage of revision numbers.
- If we build segments for non-master commits on demand, there are performance
concerns about very long release branches.
- It's very desirable to keep `ancestors(master)` continuous. In other words,
`ancestors(master)` is a single integer span. This affects performance for
DAG algorithms a lot.
To solve the problems, I'm going to use separate ranges for master and
non-master commit Ids (so assigning Ids to non-master commits wouldn't affect
what numbers can be used for the master group). To make the code easier to
read, I'm changing `type Id = u64` to `struct Id` where the struct is
group-aware.
This diff introduces related types and basic operations.
Reviewed By: sfilipco
Differential Revision: D18640899
fbshipit-source-id: 8f7ecbee6e0d1703d1225f18e92742981dbf3757
Summary:
When changing a file to a folder in a commit this function would try to query
the storage using the path and the id of the file. Storage would not have
such an entry and would cause the operation to fail.
Fixing the issue by filtering out ids of files in the section for known
manifests. Filtering already occurs in the section for transmitted manifest.
Reviewed By: quark-zju
Differential Revision: D18797368
fbshipit-source-id: 2ed727b5eaaab2f17cd988d6c8a3a1055ac8bd2a
Summary:
These are not currently used by mercurial and are superseded
by the newer more general `watchman_client` crate on crates.io:
https://docs.rs/watchman_client/0.2.0/watchman_client/
Some of the implementation of that crate was based on the work I'm
removing here.
I was going to update hg_watchman_client to be implemented in terms
of the new crate, but since it isn't used and it would only be a thin
wrapper, I figure that it is best to delete this code instead so that
we can integrate it in terms of the new API when we're ready.
Reviewed By: singhsrb
Differential Revision: D18777161
fbshipit-source-id: efacb749e6d35ff4cc1fdc7d99cdca9ed42c9cfe
Summary:
It would be nice if the insert method would give back the path that
it failed to insert.
Reviewed By: dtolnay
Differential Revision: D18739978
fbshipit-source-id: 4c16d09750ade2f01397161129c31bcf0059a957
Summary:
The Mercurial code has the bad habbit of inserting files in what can
appear impossible locations. Sometimes files with directory names and
sometimes directories in file locations. This happens because the initial
code would do additions before deletions on Manifest implementations that
were rudimentary. As soon as we introduce validation various code paths
surface.
I tried to fix the codepaths that modify manifests but it's a losing game.
I fixed the issue that appeared in tests and a couple of issues people
reported but more situations crop up.
This is giving up on the python code.
Reviewed By: quark-zju
Differential Revision: D18737678
fbshipit-source-id: 0c97128ff67e5ba2334942b6afc404aa64a2e5f4
Summary:
`path = ...` is enough. This should resolve the following build issues:
error: failed to select a version for the requirement `serde_bser = "^0.1.0"`
candidate versions found which didn't match: 0.2.0
location searched: ...
required by package `watchman_client v0.1.0 (...)
... which is depended on by `hg_watchman_client v0.1.0 (...)`
Reviewed By: lukaspiatkowski
Differential Revision: D18738176
fbshipit-source-id: 725840895a8e988b35000a48cf92018b14cb4dee
Summary:
We have seen cases on Windows where the hg process gets stuck in
CreateToolhelp32Snapshot. Let's be defensive and always exit the loop.
Reviewed By: singhsrb
Differential Revision: D18729720
fbshipit-source-id: fb8602ce231eec01b6b42c6759849d56e5db2030
Summary:
This helps test if there are uncommitted changes. It can help detecting
programming errors. For example, is metalog changed before starting a
transaction?
Reviewed By: xavierd
Differential Revision: D18538148
fbshipit-source-id: 9c82838d8174e6a93fde7108503a025660191cbf
Summary:
The current `commit` code path errors out without telling much about what
conflicts are. Improve it by showing what are actually conflicting, and allows
users to replace the resolver function to do customized merges.
Reviewed By: xavierd
Differential Revision: D18519996
fbshipit-source-id: 78f42ada1df45659783ce1eba30a19f6fc6b3b62
Summary:
Remove some unnecessary forward declarations, and use full paths in includes
so autodeps can recognize which rule these headers come from.
Reviewed By: fanzeyi
Differential Revision: D18652106
fbshipit-source-id: e2c84e1aaf846c91c18242050c4015eca8c501d3
Summary:
Now that the fb-mercurial sources are available in the Eden repository, update
our CMake build files to always build them. This moves the build logic from
the centralized `FBMercurialFeatures.cmake` file into `CMakeLists.txt` files
in the appropriate subdirectories.
Reviewed By: chadaustin
Differential Revision: D18588011
fbshipit-source-id: ded9decde5c2ec766aae0bb0f4f5b021d1044a98
Summary:
This diff replaces eden's dependencies on failure::Error with anyhow::Error.
Failure's error type requires all errors to have an implementation of failure's own failure::Fail trait in order for cause chains and backtraces to work. The necessary methods for this functionality have made their way into the standard library error trait, so modern error libraries build directly on std::error::Error rather than something like failure::Fail. Once we are no longer tied to failure 0.1's Fail trait, different parts of the codebase will be free to use any std::error::Error-based libraries they like while still working nicely together.
Reviewed By: xavierd
Differential Revision: D18576093
fbshipit-source-id: e2d862b659450f2969520d9b74877913fabb2e5d
Summary:
Add Event::Tags so we can tag commands with one or more names.
Add index for the tags so we can search them more efficiently.
As we're here, update hgcommands to read EDENSCM_BLACKBOX_TAGS and write
Event::Tags automatically just after Event::Start.
Differential Revision: D18610733
fbshipit-source-id: c69b6e3d34b317e92a1c4d76f29e832b6b01c4fa
Summary:
Previously, the blackbox index was hardcoded on the timestamp, and is not
Event-aware. This diff changes the index to parse Event and build up indexes
for various common events (ex. Start, Finish).
For now, the index just includes start, finish timestamps and duration.
Differential Revision: D18610734
fbshipit-source-id: bf572f3138c8c18d3aab06eadcc83dd5b0b9cf02
Summary:
Previously, `match_pattern` only returns a `bool`. This diff implements an
enhanced version `capture_pattern` to be able to capture matched objects.
This will be used to match query patterns and apply optimization.
Differential Revision: D18610732
fbshipit-source-id: 39e989e7633b834c04997bf8665494556f3b62b2
