Summary:
I am wondering whether we should customize the serialization format for the
InProcessStore. I want to have a basis for the comparison before I proceed.
Reviewed By: quark-zju
Differential Revision: D24580273
fbshipit-source-id: d3ddfdc029dbdd84f60acace06fddc80b4d005f4
Summary: We had some aliases for fbsource, and `fbcode` is still in use. Teach phrevset to recognise all aliases via config for ease of patching.
Reviewed By: markbt
Differential Revision: D24589906
fbshipit-source-id: bd61e86135d63ae07fa62d741e16cea4882f691b
Summary:
D24293498 (ea03d62698) is causing 'hg log scm/metagit' to fail with weird revision
number lookup errors. Let's back it out for now.
Reviewed By: singhsrb
Differential Revision: D24549796
fbshipit-source-id: b10c4caf8fdadd2955ae1f829381c66d788042e6
Summary: This is essentially a backout of D24365328 (9f664a8b30).
Reviewed By: DurhamG
Differential Revision: D24544495
fbshipit-source-id: 08f598df0892a8479fac563096f9782038e18dfe
Summary: This change adds wire types for the history API in the most straightforward way possible. In a future change, I'll move the `WireHistoryEntry` / `HistoryResponseChunk` conversion logic into the `ToApi` implementation. This implementation also doesn't add per-item errors, or standardize `history` to match the protocol evolution standards used by `trees`.
Reviewed By: kulshrax
Differential Revision: D24342046
fbshipit-source-id: d46403a823f2a1e89ad9d6d2074241d8bfe4810e
Summary: This change introduces custo `Serialize` and `Deserialize` implementations for wire types containing a fixed-length byte array. This change is more verbose than should be necessary. Because `TryFrom` is implemented to convert `&[T]` to `[T; N]` and `AsRef` to convert `[T; N]` to `&[T]`, we should be able to use a generic serde `with = ` attribute on the byte array fields of the appropriate structs. Unfortunately, I'm getting a lifetime issue that I haven't been able to resolve, and const generics aren't available on stable to implement the trait bounds without an explicit lifetime.
Reviewed By: kulshrax
Differential Revision: D24460527
fbshipit-source-id: d3a4179b833a523ce164d3df934e4cbdc2202546
Summary:
Removes top-level metadata from `TreeEntry` and add a new, specialized type for carrying child entry metadata, `TreeChildEntry`. This change also fully removes the `revisionstore_types::Metadata` from EdenAPI trees, which is only used on files.
In a follow up change I'll optimize handlings of paths / path segments in `TreeChildEntry` keys. Right now they need to be joined manually.
Reviewed By: kulshrax
Differential Revision: D24434716
fbshipit-source-id: d0739471b1f6cef58b435e10b5fb774bfb08f7f6
Summary: Rather than silently dropping entries which cannot be fetched, this change has the `WireTreeEntry` type carry optional error information, allowing it to be (de)serialized to / from `Result<TreeEntry, EdenApiServerError>` instead of a bare `TreeEntry`. Currently, handling of these failures is up to the individual application code, but it might be useful to introduce utility functions to drop failed entries and log errors.
Reviewed By: kulshrax
Differential Revision: D24315399
fbshipit-source-id: 94e4593b77cf2dc12d0dcc93d174c8a4eda95344
Summary:
`message` is not longer an attribute of an exception object in Python 3. Avoid
using it. This avoids turning rendering `{mirrornode}` template into a crash like:
Traceback (most recent call last):
File "edenscm/hgext/fbscmquery.py", line 78, in mirrornode
return client.getmirroredrev(reponame, "hg", torepo, totype, node)
File "edenscm/hgext/extlib/phabricator/graphql.py", line 399, in getmirroredrev
self._raise_errors(ret)
File "edenscm/hgext/extlib/phabricator/graphql.py", line 504, in _raise_errors
raise ClientError(None, errormsg)
edenscm.hgext.extlib.phabricator.graphql.ClientError: The provided crypto auth token(s) are expired. Check to make sure you have specified a long enough expiration time for your crypto auth token(s).
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "edenscm/mercurial/commands/__init__.py", line 4225, in log
displayer.show(
....
File "edenscm/mercurial/templatefilters.py", line 422, in byteify
return b"".join([byteify(t) for t in thing if t is not None])
....
File "edenscm/mercurial/templater.py", line 391, in evalstring
return stringify(evalrawexp(context, mapping, arg))
File "edenscm/mercurial/templater.py", line 350, in evalrawexp
return func(context, mapping, data)
File "edenscm/hgext/fbscmquery.py", line 83, in mirrornode
mapping["repo"].ui.warn(_x(str(e.message) + "\n"))
AttributeError: 'ClientError' object has no attribute 'message'
Reported by: arnabde03
Reviewed By: mofarrell
Differential Revision: D24528362
fbshipit-source-id: 57d093811986f3a65d1201de1f9aed3770db2ad1
Summary:
Users get confused and try to pull from the wrong repo - e.g. `hg checkout D23665606` in `fbsource` - and find the resulting output about missing hashes scary.
Give them a nice clear error instead.
Reviewed By: wez
Differential Revision: D24485097
fbshipit-source-id: e9aeb7ed6aaf494f828d736ca5203100807fb321
Summary:
indexedlogdatastore is supposed to use remotefilelog.cachelimit to set
the max size, but instead it was setting the max per-log size, which means the
max size was N times bigger. Let's fix that.
Reviewed By: xavierd
Differential Revision: D24483181
fbshipit-source-id: f33cedbfdbb318e9d5eb9fda497645050b93e9fe
Summary:
In the next diff in this stack, I'm changing the Rust thrift library as well as
the codegen, and this is causing quite a bit of pain with regard to this
codegen that is checked in to the hg build:
- Regenerating the codegen isn't super obvious.
- The Sandcastle build fails because it uses the current codegen with the old
library.
According to lukaspiatkowski, this has also been a problem in the past when Eden got
migrated to Tokio 0.2, so I'd like to save myself and others some pain by just
not generating the server side codegen and not checking it in, since it's
unused. This reduces the surface of stuff that might go out of sync.
#forcetdhashing
Reviewed By: markbt
Differential Revision: D23649604
fbshipit-source-id: d684bec427431a366de42c88e53072caa98d5b2f
Summary:
When the new rust store is enabled, it can throw exceptions if the rust
code tries to fetch multiple trees when the server doesn't support bfs fetching.
The python code path ignores these requests, letting the eventual single tree
lookup path (i.e. not the prefetch path) perform the download. Let's let the
rust code do the same for rust stores.
Eventually every repo will support bfs fetching and we can delete this code path
entirely.
Reviewed By: quark-zju
Differential Revision: D24459106
fbshipit-source-id: d9a931364bd3235113b0dbcf09cba35a16e16e8e
Summary:
Use the new API `smartset.prefetch` for the prefetching logic for `log`
commands.
Prefetching is only enabled if the template contains related symbols.
Reviewed By: DurhamG
Differential Revision: D24289052
fbshipit-source-id: 810da3b249d45a7516aaf39a0b74f6965b87f96f
Summary: This makes `log` command more efficient using the `hybrid` backend.
Reviewed By: DurhamG
Differential Revision: D24293498
fbshipit-source-id: db3bac28a0b04ba6af449357bc08b285cd62780d
Summary:
Add prefetch APIs on smartset to make it easier to specify what need to be
prefetched.
This is intended to be used by `{phabstatus}`, log with lazy commit text backend,
and revsets like `author(x)`, etc.
Reviewed By: sfilipco
Differential Revision: D24321292
fbshipit-source-id: 7e645ccbe39feeae1e2da7bfb115623945bf9cef
Summary:
This makes it possible to "fork" sets and set different prefetch properties on
them.
Reviewed By: sfilipco
Differential Revision: D24365329
fbshipit-source-id: c4bdd6ab21e6a891a02d450f9e6edab3a3781eca
Summary:
The generatorset has a pure Python implementation for rewindable generator.
However it is not thread-safe, and we want thread-safety since the set
iteration will be driven by async Rust in multiple threads.
One of the issues is, during thread 1 `next(gen)`, thread 2 might call
`next(gen)`, and that's not allowed by Python.
Fix them by switching to the Rust RGenerator.
Reviewed By: DurhamG
Differential Revision: D24365328
fbshipit-source-id: 2785e80c7c460a7f754ed23e3af99f4a5c9fbcdf
Summary:
Similarly to the indexedlogdatastore, the LFS stores can become corrupted on
power loss. This is happening fairly frequently on Windows Sandcastle due to
the OS being virtualized and power being cut abruptly.
For now this only attempts to repair the shared stores, in theory we could also
try to repair the local stores but haven't looked into it.
Reviewed By: DurhamG
Differential Revision: D24449202
fbshipit-source-id: 605a7943a0850b625bf00c514879b3da1ab2b406
Summary:
The ContentStore/Metadatastore are made of several different stores, attempting
to expose all of them to Python to drive the repair logic from there would leak
implementation detail of how the stores are implemented.
Instead, let's simply expose a single `repair` function out of the
pyrevisionstore crate that takes care of repairing all of the underlying
stores. For now, this is just moving code around, but a future diff will
integrate the LFS stores.
Reviewed By: DurhamG
Differential Revision: D24449203
fbshipit-source-id: 1631ced9068716453cb404bf7e65cefbf2db5247
Summary:
This will be used to avoid 1-by-1 fetching for the changelog backend with
commit text stored remotely.
Reviewed By: sfilipco
Differential Revision: D24321293
fbshipit-source-id: 9695c72166cadc0b167e2ce7fde822cdf6b1cea8
Summary: This will be used to test if a keyword is in an AST.
Reviewed By: DurhamG
Differential Revision: D24293499
fbshipit-source-id: c4c238e09fae6728d9d20ffe94e04194ac20bbb8
Summary:
There were 2 recent reports about "visibleheads" not being enabled.
Previously, to enable visibility, one has to run a `pull` command with
`visibility.automirate=start` and `visibility.enabled=true`, and
`pull.automirate=true`, and our current config does not have
`visibility.automigrate`.
There are other issues with pull-based auto migration. For example, the edenfs
backing repo does not run pull and won't be migrated.
Move the visibility and dirstate automigrate logic to `repo.__init__` so they
always run. Also simplify the configuration so a single config
`visibility.enabled=true` (or `format.dirstate`) is enough to trigger the
migration.
Reviewed By: DurhamG
Differential Revision: D24340586
fbshipit-source-id: 2b8ed8da51c2345c9e6e680519dbc4f10eab1ac0
Summary:
We expect `.hg/store` to exist (ex. `storerequirements`). Drop support for repo
layout without a store.
Reviewed By: DurhamG
Differential Revision: D24340585
fbshipit-source-id: 757dc8633db9549a6bea082281326ebdefcbfcba
Summary:
Treestate is the only supported dirstate format. Remove test cases about legacy
dirstates.
Reviewed By: DurhamG
Differential Revision: D24340584
fbshipit-source-id: 2ebaa7d5aa9e8639e3ce4caa67c4f93155b52558
Summary:
Turn on rust changelog (changelog2) for all hosts (except hgsql).
Turn on doublewrite backend for hg-dev hosts, triggered by pull.
Tests are mostly working, and I have been using it for weeks.
Reviewed By: singhsrb
Differential Revision: D24259759
fbshipit-source-id: b89a27f98a6d3d1e4ea187bf7b29f875d0e96e2e
Summary: It will be used to test whether `{phabstatus}` is in template.
Reviewed By: sfilipco
Differential Revision: D24293496
fbshipit-source-id: 1b59236e07280c0cefb07be9da7a3fd18d9940a1
Summary:
There are 2 issues:
- len(repo) is no longer the "max rev" + 1. Use 1<<63 instead.
- If invalidatelinkrev is a repo requirement, linkrev can no longer be trusted.
Reviewed By: DurhamG
Differential Revision: D24270423
fbshipit-source-id: 4e5c999772fad6ef2b1dabd2d6363d4226242c4d
Summary:
If linkrevs are known unreliable, do not use them. This fixes some cases with
segmented changelog, file revisions are incorrectly filtered out and not sent
to client.
Reviewed By: DurhamG
Differential Revision: D24233228
fbshipit-source-id: 268a95b44c0217658b849317cb0a680450b91d23
Summary: This makes the output less verbose.
Reviewed By: DurhamG
Differential Revision: D24411259
fbshipit-source-id: 50b175f2e31ffda62e78dd226775edaf53893bce
Summary:
With segmented changelog, the maximum revision is not always the "tip".
Therefore, track "tip" explicitly in svfs / metalog. This improves
compatibility.
Reviewed By: DurhamG
Differential Revision: D24214711
fbshipit-source-id: 7b22de7ac684a90965a695924bdb9114758aae43
Summary: Avoid HashSet or HashMap order to preserve the order of inserting commits.
Reviewed By: DurhamG
Differential Revision: D24214460
fbshipit-source-id: 66df2e0aba1820e6585f8da66897078f38abf82f
Summary:
Previously bundlerepo only supports revlog changelog. Detect repo configuration
and support non-revlog changelog. This is done by adding commits from bundle in
memory, and updating visibleheads.
Reviewed By: DurhamG
Differential Revision: D24214461
fbshipit-source-id: 63f9af49a0fe0ed2581b611f3cf7002d1b89d2fd
Summary:
The nullid and wdirid are special hg hashes that do not respect SHA1. They were
handled at the bindings layer. However the bindings layer cannot handle them
in a stream. Therefore move it in hgcommits.
Reviewed By: DurhamG
Differential Revision: D24365330
fbshipit-source-id: e8dc6205351ec1a2304252b9ec446dda010e6295
Summary:
Pure Python generator has a few limitations:
- Can only be iterated once.
- Cannot be iterated by multiple threads concurrently.
By converting revset iterators to Rust Stream, they run in different threads
and can context switch in `next()`, breaking the program.
Provide a native reentrant generator implementation to address the issue.
This is sound because Rust methods cannot be interrupted by Python.
Reviewed By: DurhamG
Differential Revision: D24365331
fbshipit-source-id: 885dade922b7863a73203b206a96b492d55bccd0
Summary:
Update the return type of streamcommitrawtext from
`{"vertex": v, "raw_text": t}` to `(v, t)`. This makes it easier to use
in Python, as Python supports `for v, t in ...` but does not support
`for {"vertex": v, "raw_text": t} in ...`.
Reviewed By: DurhamG
Differential Revision: D24295457
fbshipit-source-id: 284a29b9deae2d8509d3afea0fcbcaadbfebbae8
Summary:
In case the server does not respect the input contract and missed
some items without returning errors. The current logic would retry
forever. Change it to detect the issue and raise an error.
Reviewed By: DurhamG
Differential Revision: D24293497
fbshipit-source-id: 09421c7743078a488a9c81ce66fd92c12b39543c
Summary: The store stores sorted(p1,p2)+text to match SHA1 hashes. It's not just `text`.
Reviewed By: DurhamG
Differential Revision: D24325554
fbshipit-source-id: 8a91970f60fb535ca1a5a2d30c7d27f2714f28de
Summary: It's no longer useful as the new abstract interface does not need it.
Reviewed By: sfilipco
Differential Revision: D24399516
fbshipit-source-id: 2b6735d2a26706c6a3e6b592d2f3ecfc874c94cb
Summary:
This verifies the abstraction and simplifies the code.
The new code will use non-master segments for add_heads. Therefore the test
changes.
Reviewed By: sfilipco
Differential Revision: D24399496
fbshipit-source-id: 39067ad88ade79b4f7758bcdaafc03e5f34ced91
