Summary:
This function is useful in the mononoke to compute the universal commit idmap
that is required for clone.
Reviewed By: quark-zju
Differential Revision: D24808327
fbshipit-source-id: 0cccd59bd7982dd0bc024d5fc85fb5aa5eafb831
Summary:
`flat_segments` are going to be used to generate CloneData. These segments will
be sent to a client repository and are going to bootstrap the iddag.
Reviewed By: quark-zju
Differential Revision: D24808331
fbshipit-source-id: 00bf9723a43bb159cd98304c2c4c6583988d75aa
Summary: This is the object that will be used to bootstrap a Dag after a clone.
Reviewed By: quark-zju
Differential Revision: D24808328
fbshipit-source-id: 2c7e97c027c84a11e8716f2e288500474990169b
Summary:
The goal is to reused the functionality provided by AssignHeadOutcome for clone
purposes.
Reviewed By: quark-zju
Differential Revision: D24717924
fbshipit-source-id: e88f21ee0d8210e805e9d6896bc8992009bd7975
Summary:
The original problem was a fastlog bug, solved by D24513444 (c3bcc1ab88).
Restores prefetching for phabricator status so `hg ssl` and `hg fssl` become fast again.
Original commit changeset: b10c4caf8fda
Reviewed By: sfilipco
Differential Revision: D24749774
fbshipit-source-id: fa14f7dde9c922733525a7ff014efc32875426fa
Summary:
The original issue was a rust-cpython bug, solved by D24698226, or https://github.com/dgrunwald/rust-cpython/pull/244.
Original commit changeset: 08f598df0892
Reviewed By: sfilipco
Differential Revision: D24759765
fbshipit-source-id: f9a1359cfce68c8754ddd1bcb8bfc54bf75af7ff
Summary:
On Windows terminal, with light color schemes, crecord text was barely visible
(sometimes invisible) due to low contrast on either the background, or the
foreground. Making the text bold makes it brighter and thus more readable.
As a bonus, I've also made the hunk lines magenta to mimic what `hg diff` does.
Reviewed By: DurhamG
Differential Revision: D24718598
fbshipit-source-id: 18c2ff03fc2a46ca45808d5061db21e1f1b501ae
Summary: This makes it clean up stale files more aggressively.
Reviewed By: DurhamG
Differential Revision: D24744461
fbshipit-source-id: 76d163c9f16d8f8d1bf628e9197a3086d7cd48aa
Summary:
The goal of this code was to divide the cache limit by the number of
logs. Instead it divided the cache limit by the default per-log size (2GB). That
results in a very small max-bytes-per-log so data was being thrown out
constantly. This fixes it and updates tests to actually demonstrate the issue.
Reviewed By: kulshrax
Differential Revision: D24712842
fbshipit-source-id: 8062758b5bfa40493e2003d5a9028d601b1522b1
Summary:
Python 3 doesn't support line buffering for binary file descriptors.
Let's stop setting it in chg.
This was causing warnings to pop up during prompts for users.
```
.../python3.8/os.py:1023: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
return io.open(fd, *args, **kwargs)
```
Reviewed By: singhsrb
Differential Revision: D24747777
fbshipit-source-id: 0b881b4067e8c7086fe73380f81d526a2ecc364a
Summary:
Downloading and applying mercurial bundles directly for list of given heads.
Backup store stores commits as mercurial bundles that can be fetched directly from the store and applied (everstore).
The command could be useful when we migrate our server from one backend to another (Mononoke) and some commits can be missing in Mononoke.
The command could probably be deleted after a while once we migrate completely...
Reviewed By: mitrandir77
Differential Revision: D24756583
fbshipit-source-id: 1629c3756f244621efb965dfe15b75c7509a1cd1
Summary:
This diff makes "Calculating additional actions for sparse profile update" more
efficient by using xormatcher instead of unionmatcher. Indeed, we are
interested only in files that changed their "state" after sparse profile change
e.g. either a file was included in sparse profile and then became excluded.
Reviewed By: sfilipco
Differential Revision: D24725902
fbshipit-source-id: ee611e7c123b95937652ced828b5bea6d75a3daf
Summary:
At the moment differencematcher.visitdir never returns "all".
This diff changes it to return all in the case if self._m2 doesn't visit the directory at all and
self.m1.visitdir(dir) returns "all". This makes sense - if m1 visits all files
in the directory and m2 doesn't exclude any file then it's safe to return all
in this case.
This optimization will be used in the next diff.
Reviewed By: sfilipco
Differential Revision: D24725903
fbshipit-source-id: 2a049cfb1ea4878331e8640cbb20af74da86a1a1
Summary:
Whenever a sparse profile changes (e.g. we include or exclude a directory or a file) we do a full prefetch for all trees in the revision and then for each file in a revision we check if this file has changed its state after sparse profile change (i.e. whether it was included before the change and became excluded after the change and vice versa). It can be quite expensive for large repos and looks like checking all the files is unnecessary.
For example, there might be top-level directories that are excluded in sparse profile before and after the change. In that case there's no reason to check every file in this directory, and there's no reason to prefetch manifests for this directory.
More importantly, `mf.walk()` method is already smart enough to do manifest prefetches if treemanifest.ondemandfetch is set to True, so it looks like there's no reason to do any additional prefetching at all (at least in theory).
So this diff does a few things:
1) The default mode is to use mf.walk() method with a union matcher to find all the files that were are included either in old or new sparse profile. In order for it to prefetch efficiently we force enable treemanifest.ondemandfetch config option.
2) It also adds a fallback option to full prefetch (i.e. the same thing we do right now) Hopefully this fallback option won't be necessary and we'll delete them soon. I've added them only to be able to fallback to current behaviour in case there are problems with the new behaviour
I think we can do an even more efficient fetch by using xor matcher instead of union matcher. I'll try to implement it in the next diffs
Reviewed By: sfilipco
Differential Revision: D24705823
fbshipit-source-id: 2c232a66cc74ee95bdaa84201df46448412f087f
Summary:
The revlog changelog has incompatible rev numbers with changelog2 backends. Do
not construct it. Instead, just use the current changelog.
Reviewed By: DurhamG
Differential Revision: D24513444
fbshipit-source-id: 35d9326cd9fde4af8b98d628f6df66bd80883f92
Summary:
The goal of this diff is to provide more visibility into how long the client
takes to create/upload an infinitepush bundle. This is done in two ways:
- by adding more `perftrace` calls (useful when invistigating individual slow
pushes)
- by adding `ui.timesection` scopes (useful for aggregation purposes)
Two main things that are measured:
- creation of the bundle purely on the client
- sending of the bundle over the wire
In addition, in the perftrace recording, this measures how long it takes to
process the reply handlers, how much bytes are sent over the wire, what are the
part names and sizes (when available). These changes mostly do not distinguish
whether the code is infinitepush push or not, but they are always related to
some sort of a wireproto scenario, which means that the performance impact is
negligible (writing things to thread-local storage is *much* cheaper than
sending them over the network).
Reviewed By: DurhamG
Differential Revision: D24683484
fbshipit-source-id: 53fdfb63dcdfcf38924237c59a1e8f5e24ff96c0
Summary: We were incorrectly marking reverts as landed during pullcreatemarkers.
Reviewed By: quark-zju
Differential Revision: D24608217
fbshipit-source-id: f919f49469d6933c17894b3b0926ba2430a5947a
Summary:
As part of getting buck build to work on OSX, we need procinfo to
include it's OSX specific library.
Reviewed By: sfilipco
Differential Revision: D24513234
fbshipit-source-id: 69d8dd546e28b4403718351ff7984ee6b2ed3d1d
Summary:
Improve ux for setting the token.
This extra bits would help to figure out prompt issues on Windows
Reviewed By: markbt
Differential Revision: D24706085
fbshipit-source-id: 6101b8d7b90aad2d687465a09cc69670ca4a46f6
Summary:
In Mercurial we're finding use cases for conditioning on buck vs
non-buck builds. For instance, in some cases we specify #[link] directives for
non-buck builds, but don't need to for buck builds
(https://fb.workplace.com/groups/rust.language/permalink/4568487309866515/). In
other cases we want to depend on internal buck libraries when building with
buck, and not otherwise.
Let's add a fb_buck_build cfg so other projects may distinguish the same.
Reviewed By: jsgf
Differential Revision: D24493974
fbshipit-source-id: 1d558cbe0ae01ab4a7b4b5d6d4be75bb8ab0f41a
Summary:
The rage local config output was getting polluted by dynamic config.
Let's filter them out.
Reviewed By: farnz
Differential Revision: D24626564
fbshipit-source-id: df5ac04cd549595ecdccc0b2438d4e7c72b80e88
Summary:
Previously, reading from SSH subprocesses did’t return on Python 3. bufsize=0
is default on Python 2, but not Python 3.
This is a backport of: 0fc8020ebe
Reviewed By: krallin
Differential Revision: D24652418
fbshipit-source-id: 1140c76b6f711bfe1726108bd4fe6948e6ee41a0
Summary:
I initially saw the incremental build as something that would be run in places
that had IdMap and IdDag stored side by side in process. I am reconsidering
to use incremental build in the tailing process to keeps Segmented Changelog
artifacts up to date.
Since we update the IdMap before we update the IdDag, it is likely that we
will have runs that only update the IdMap and fail to update IdDags. This diff
adds a mechanism for the IdDag to catch up.
Reviewed By: krallin
Differential Revision: D24516440
fbshipit-source-id: 3a99248451d806ae20a0ba96199a34a8a35edaa4
Summary:
mitrandir77 pointed me to this field which is exactly what I want.
Leave alias handling in place for configerator, which has a different name.
Reviewed By: mitrandir77
Differential Revision: D24628323
fbshipit-source-id: ccd01e6314feefd8ab75e8052a0a8a98b3119691
Summary: This diff removes reading the token from the "cert" property in the `~/.arcrc` and forces the caller to use either an OAuth token or CATs.
Reviewed By: quark-zju
Differential Revision: D24242614
fbshipit-source-id: 18538270102b7aa28731e82c8dd21f5da9e2f2d6
Summary:
We have automation that wants to use this to hold the lock while doing
some maintenance. They want the ability to wait for the lock so they don't have
to busy loop.
Reviewed By: snarkmaster
Differential Revision: D24604466
fbshipit-source-id: be02539908655e183f334865718b68b633b069a5
Summary:
I am wondering whether we should customize the serialization format for the
InProcessStore. I want to have a basis for the comparison before I proceed.
Reviewed By: quark-zju
Differential Revision: D24580273
fbshipit-source-id: d3ddfdc029dbdd84f60acace06fddc80b4d005f4
Summary: We had some aliases for fbsource, and `fbcode` is still in use. Teach phrevset to recognise all aliases via config for ease of patching.
Reviewed By: markbt
Differential Revision: D24589906
fbshipit-source-id: bd61e86135d63ae07fa62d741e16cea4882f691b
Summary:
D24293498 (ea03d62698) is causing 'hg log scm/metagit' to fail with weird revision
number lookup errors. Let's back it out for now.
Reviewed By: singhsrb
Differential Revision: D24549796
fbshipit-source-id: b10c4caf8fdadd2955ae1f829381c66d788042e6
Summary: This is essentially a backout of D24365328 (9f664a8b30).
Reviewed By: DurhamG
Differential Revision: D24544495
fbshipit-source-id: 08f598df0892a8479fac563096f9782038e18dfe
Summary: This change adds wire types for the history API in the most straightforward way possible. In a future change, I'll move the `WireHistoryEntry` / `HistoryResponseChunk` conversion logic into the `ToApi` implementation. This implementation also doesn't add per-item errors, or standardize `history` to match the protocol evolution standards used by `trees`.
Reviewed By: kulshrax
Differential Revision: D24342046
fbshipit-source-id: d46403a823f2a1e89ad9d6d2074241d8bfe4810e
Summary: This change introduces custo `Serialize` and `Deserialize` implementations for wire types containing a fixed-length byte array. This change is more verbose than should be necessary. Because `TryFrom` is implemented to convert `&[T]` to `[T; N]` and `AsRef` to convert `[T; N]` to `&[T]`, we should be able to use a generic serde `with = ` attribute on the byte array fields of the appropriate structs. Unfortunately, I'm getting a lifetime issue that I haven't been able to resolve, and const generics aren't available on stable to implement the trait bounds without an explicit lifetime.
Reviewed By: kulshrax
Differential Revision: D24460527
fbshipit-source-id: d3a4179b833a523ce164d3df934e4cbdc2202546
Summary:
Removes top-level metadata from `TreeEntry` and add a new, specialized type for carrying child entry metadata, `TreeChildEntry`. This change also fully removes the `revisionstore_types::Metadata` from EdenAPI trees, which is only used on files.
In a follow up change I'll optimize handlings of paths / path segments in `TreeChildEntry` keys. Right now they need to be joined manually.
Reviewed By: kulshrax
Differential Revision: D24434716
fbshipit-source-id: d0739471b1f6cef58b435e10b5fb774bfb08f7f6
Summary: Rather than silently dropping entries which cannot be fetched, this change has the `WireTreeEntry` type carry optional error information, allowing it to be (de)serialized to / from `Result<TreeEntry, EdenApiServerError>` instead of a bare `TreeEntry`. Currently, handling of these failures is up to the individual application code, but it might be useful to introduce utility functions to drop failed entries and log errors.
Reviewed By: kulshrax
Differential Revision: D24315399
fbshipit-source-id: 94e4593b77cf2dc12d0dcc93d174c8a4eda95344
Summary:
`message` is not longer an attribute of an exception object in Python 3. Avoid
using it. This avoids turning rendering `{mirrornode}` template into a crash like:
Traceback (most recent call last):
File "edenscm/hgext/fbscmquery.py", line 78, in mirrornode
return client.getmirroredrev(reponame, "hg", torepo, totype, node)
File "edenscm/hgext/extlib/phabricator/graphql.py", line 399, in getmirroredrev
self._raise_errors(ret)
File "edenscm/hgext/extlib/phabricator/graphql.py", line 504, in _raise_errors
raise ClientError(None, errormsg)
edenscm.hgext.extlib.phabricator.graphql.ClientError: The provided crypto auth token(s) are expired. Check to make sure you have specified a long enough expiration time for your crypto auth token(s).
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "edenscm/mercurial/commands/__init__.py", line 4225, in log
displayer.show(
....
File "edenscm/mercurial/templatefilters.py", line 422, in byteify
return b"".join([byteify(t) for t in thing if t is not None])
....
File "edenscm/mercurial/templater.py", line 391, in evalstring
return stringify(evalrawexp(context, mapping, arg))
File "edenscm/mercurial/templater.py", line 350, in evalrawexp
return func(context, mapping, data)
File "edenscm/hgext/fbscmquery.py", line 83, in mirrornode
mapping["repo"].ui.warn(_x(str(e.message) + "\n"))
AttributeError: 'ClientError' object has no attribute 'message'
Reported by: arnabde03
Reviewed By: mofarrell
Differential Revision: D24528362
fbshipit-source-id: 57d093811986f3a65d1201de1f9aed3770db2ad1
Summary:
Users get confused and try to pull from the wrong repo - e.g. `hg checkout D23665606` in `fbsource` - and find the resulting output about missing hashes scary.
Give them a nice clear error instead.
Reviewed By: wez
Differential Revision: D24485097
fbshipit-source-id: e9aeb7ed6aaf494f828d736ca5203100807fb321
Summary:
indexedlogdatastore is supposed to use remotefilelog.cachelimit to set
the max size, but instead it was setting the max per-log size, which means the
max size was N times bigger. Let's fix that.
Reviewed By: xavierd
Differential Revision: D24483181
fbshipit-source-id: f33cedbfdbb318e9d5eb9fda497645050b93e9fe
Summary:
In the next diff in this stack, I'm changing the Rust thrift library as well as
the codegen, and this is causing quite a bit of pain with regard to this
codegen that is checked in to the hg build:
- Regenerating the codegen isn't super obvious.
- The Sandcastle build fails because it uses the current codegen with the old
library.
According to lukaspiatkowski, this has also been a problem in the past when Eden got
migrated to Tokio 0.2, so I'd like to save myself and others some pain by just
not generating the server side codegen and not checking it in, since it's
unused. This reduces the surface of stuff that might go out of sync.
#forcetdhashing
Reviewed By: markbt
Differential Revision: D23649604
fbshipit-source-id: d684bec427431a366de42c88e53072caa98d5b2f
Summary:
When the new rust store is enabled, it can throw exceptions if the rust
code tries to fetch multiple trees when the server doesn't support bfs fetching.
The python code path ignores these requests, letting the eventual single tree
lookup path (i.e. not the prefetch path) perform the download. Let's let the
rust code do the same for rust stores.
Eventually every repo will support bfs fetching and we can delete this code path
entirely.
Reviewed By: quark-zju
Differential Revision: D24459106
fbshipit-source-id: d9a931364bd3235113b0dbcf09cba35a16e16e8e
Summary:
Use the new API `smartset.prefetch` for the prefetching logic for `log`
commands.
Prefetching is only enabled if the template contains related symbols.
Reviewed By: DurhamG
Differential Revision: D24289052
fbshipit-source-id: 810da3b249d45a7516aaf39a0b74f6965b87f96f
Summary: This makes `log` command more efficient using the `hybrid` backend.
Reviewed By: DurhamG
Differential Revision: D24293498
fbshipit-source-id: db3bac28a0b04ba6af449357bc08b285cd62780d
Summary:
Add prefetch APIs on smartset to make it easier to specify what need to be
prefetched.
This is intended to be used by `{phabstatus}`, log with lazy commit text backend,
and revsets like `author(x)`, etc.
Reviewed By: sfilipco
Differential Revision: D24321292
fbshipit-source-id: 7e645ccbe39feeae1e2da7bfb115623945bf9cef
Summary:
This makes it possible to "fork" sets and set different prefetch properties on
them.
Reviewed By: sfilipco
Differential Revision: D24365329
fbshipit-source-id: c4bdd6ab21e6a891a02d450f9e6edab3a3781eca
Summary:
The generatorset has a pure Python implementation for rewindable generator.
However it is not thread-safe, and we want thread-safety since the set
iteration will be driven by async Rust in multiple threads.
One of the issues is, during thread 1 `next(gen)`, thread 2 might call
`next(gen)`, and that's not allowed by Python.
Fix them by switching to the Rust RGenerator.
Reviewed By: DurhamG
Differential Revision: D24365328
fbshipit-source-id: 2785e80c7c460a7f754ed23e3af99f4a5c9fbcdf
Summary:
Similarly to the indexedlogdatastore, the LFS stores can become corrupted on
power loss. This is happening fairly frequently on Windows Sandcastle due to
the OS being virtualized and power being cut abruptly.
For now this only attempts to repair the shared stores, in theory we could also
try to repair the local stores but haven't looked into it.
Reviewed By: DurhamG
Differential Revision: D24449202
fbshipit-source-id: 605a7943a0850b625bf00c514879b3da1ab2b406
Summary:
The ContentStore/Metadatastore are made of several different stores, attempting
to expose all of them to Python to drive the repair logic from there would leak
implementation detail of how the stores are implemented.
Instead, let's simply expose a single `repair` function out of the
pyrevisionstore crate that takes care of repairing all of the underlying
stores. For now, this is just moving code around, but a future diff will
integrate the LFS stores.
Reviewed By: DurhamG
Differential Revision: D24449203
fbshipit-source-id: 1631ced9068716453cb404bf7e65cefbf2db5247
Summary:
This will be used to avoid 1-by-1 fetching for the changelog backend with
commit text stored remotely.
Reviewed By: sfilipco
Differential Revision: D24321293
fbshipit-source-id: 9695c72166cadc0b167e2ce7fde822cdf6b1cea8
