Summary: Previously, we would only allow BFS prefetching when HTTP was enabled (since prior to the `designatednodes` server capability, SSH had no way of supporting BFS). That has now changed, so we should allow BFS even if HTTP is disabled.
Reviewed By: quark-zju
Differential Revision: D21581215
fbshipit-source-id: b9d5945730e2c718681004f7d5e5f319449fcf6a
Summary:
The latest version of Black removes unneccessary parenthesis. Mercurial's
test-check-code currently uses extra parentheses to signal untranslated strings, so
Black's reformatting breaks this test.
Capitulate to Black by adding a new `_x` translation marker that means "untranslated".
Reviewed By: quark-zju
Differential Revision: D21426335
fbshipit-source-id: a6c26d7c6365c49530a7dee3a5f9ed71ff166835
Summary: This will help us debug slow commands
Reviewed By: xavierd
Differential Revision: D21075895
fbshipit-source-id: 3e7667bb0e4426d743841d8fda00fa4a315f0120
Summary:
Purging treemanifest packfiles always made little sense to me, as the reason we
have tons of them, is because we fetched tons of them. Remove all of them would
just lead to the same situation again. Let's just do a repack instead.
Reviewed By: markbt
Differential Revision: D21065723
fbshipit-source-id: a7cdee1de1d7897481f75c13d8aceeb8c1a68550
Summary: In Python3, str is not a byte string, but a unicode one. Use bytes instead.
Reviewed By: sfilipco
Differential Revision: D20965794
fbshipit-source-id: b04ff004aac87fd3afc8070fc8d1d849ad48d0d3
Summary:
Similar to D19957251, failing to prefetch trees post-pull is not a fatal error.
This recently became an issue because with these conditions:
- We're migrating a repo from hg to Mononoke, and Mononoke has lagged
bookmarks, esp. remote/master.
- We're still using Mercurial to serve infinitepush pulls (ex. scratch
bookmarks). Mercurial will return bookmarks that are not synced to
Mononoke yet.
- I made a recent change (D20531121) that moves selectivepull logic that
changes `pull -r HASH` to `pull -r HASH -B master` to core. The CI
tier didn't have selectivepull enabled that will pull bookmarks, but
now they also pull bookmarks.
- The tree prefetch logic uses unpatched paths.default to fetch `master`,
which can be an unknown commit for Mononoke.
Ideally, the final state is to rely on Mononoke for everything. There is
only `paths.default` which points to Mononoke. For now, as a temporary
fix let's just make post-pull tree prefetch non-fatal. Once this happens,
it will be logged to the cloud table with fatal=false.
Reviewed By: krallin
Differential Revision: D20873599
fbshipit-source-id: 291a2dc41c7c602f004ad8df71e19a48a98e9c5e
Summary:
One of the main drawback of the current version of repack is that it writes
back the data to a packfile, making it hard to change file format. Currently, 2
file format changes are ongoing: moving away from packfiles entirely, and
moving from having LFS pointers stored in the packfiles, to a separate storage.
While an ad-hoc solution could be designed for this purpose, repack can
fullfill this goal easily by simply writing to the ContentStore, the
configuration of the ContentStore will then decide where this data will
be written into.
The main drawback of this code is the unfortunate added duplication of code.
I'm sure there is a way to avoid it by having new traits, I decided against it
for now from a code readability point of view.
Reviewed By: DurhamG
Differential Revision: D20567118
fbshipit-source-id: d67282dae31db93739e50f8cc64f9ecce92d2d30
Summary: Based on comments on D20382825, we need to make sure that `_gettrees()` knows for sure whether on-demand tree fetching is in use in order to properly identify missing nodes in the response.
Reviewed By: quark-zju
Differential Revision: D20520439
fbshipit-source-id: ffa6d62dbe8b6f641b1dacebcb6f94ceae714c1b
Summary:
This diff adds client-side support for the new `designatednodes` mode of the `gettreepack` wireproto command (implemented in D20307442), which allows fetching an exact set of specified tree nodes. This enables the performance benefits of BFS tree fetching over SSH (whereas this sort of fetching was previously only possible over HTTPS via EdenAPI).
To allow us to smoothly control the rollout of this feature, it is gated behind the `treemanifest.ondemandfetch` config option. In theory, this should not be needed since the client will check that the server supports `designatednodes` before attempting to do on-demand fetching, but if users report issues it's nice to have a killswitch.
Reviewed By: quark-zju
Differential Revision: D20382825
fbshipit-source-id: f9aef3fdae08ae84a9720deead3549fe2fba0b9b
Summary:
We cannot differentiate an empty list from a list containing the empty path only. This fixes that.
See D20309700 earlier in this stack for the server-side bits.
Reviewed By: quark-zju
Differential Revision: D20309699
fbshipit-source-id: 02efc747c3c5e2774f949ba0a6370169b55f3cf3
Summary:
This module previously used to handle deciding how a particular module should
be imported if it had multiple versions (e.g., pure Python or native).
However, as of D18819680 it was changed to always import the native C version.
Lets go ahead and remove it entirely now. Using `policy.importmod` simply
makes it harder for type checkers to figure out the actual module that will be
used.
The only functionality that `policy.importmod()` still provided was verifying
that the module contained a "version" field that looked like what was
expected. In practice these version numbers are not bumped often, so this
doesn't really seem to provide much value in checking that we imported the
correct version that we expected to be shipped with this release.
Reviewed By: xavierd
Differential Revision: D19958227
fbshipit-source-id: 05f1d027d0a41cf99c4aa93cb84a51e830305077
Summary:
Not being able to prefetch draft parent trees should not be considered as a
fatal error.
This code path is causing trouble with narrow-heads clone:
1. Streaming clone. The client gets a changelog.
2. The client runs "pull" to get new commits. The prefetchdraftparents code path runs.
3. The client has stale remote names, and public() is lagging. `prefetchdraftparents`
will try to fetch trees at the old master, but the repo is not configured properly.
That causes a stacktrace like:
$ /usr/bin/hg --config 'extensions.fsmonitor=!' clone --shallow -U --config 'ui.ssh=ssh -oControlMaster=no' --configfile /etc/mercurial/repo-specific/www.rc ssh://hg.fb.com/repo repo
connected to hg.fb.com
streaming all changes
searching for changes
adding commits
adding manifests
adding file changes
added 1 commits with 0 changes to 0 files # <<<< No traceback if this says "0 commit".
Traceback (most recent call last):
File "edenscm/hgext/remotenames.py", line 1464, in exclonecmd
orig(ui, *args, **opts)
File "edenscm/hgext/remotefilelog/__init__.py", line 433, in cloneshallow
orig(ui, repo, *args, **opts)
File "edenscm/mercurial/commands/__init__.py", line 1615, in clone
shareopts=shareopts,
# shareopts = {'mode': 'identity'}
File "edenscm/mercurial/hg.py", line 741, in clone
exchange.pull(local, srcpeer, revs, streamclonerequested=stream)
File "edenscm/mercurial/util.py", line 621, in __exit__
self.close()
File "edenscm/mercurial/transaction.py", line 46, in _active
return func(self, *args, **kwds)
File "edenscm/mercurial/transaction.py", line 543, in close
self._postclosecallback[cat](self)
# cat = bin('6472616674706172656e74747265656665746368')
File "edenscm/hgext/treemanifest/__init__.py", line 490, in _parenttreefetch
self.prefetchtrees([c.manifestnode() for c in draftparents])
# c = <changectx b5ad643b3009>
# draftparents = [<changectx b5ad643b3009>]
File "edenscm/hgext/treemanifest/__init__.py", line 522, in prefetchtrees
self._prefetchtrees("", mfnodes, basemfnodes, [], depth)
# basemfnodes = [bin('a25f17018d7cd07f1f6bc3076f95c5980ba087a9')]
# mfnodes = [bin('ad717aac7700e783a1d84f3330d13a7731a4726a')]
File "edenscm/hgext/treemanifest/__init__.py", line 529, in _prefetchtrees
fallbackpath = getfallbackpath(self)
File "edenscm/hgext/treemanifest/__init__.py", line 2173, in getfallbackpath
if util.safehasattr(repo, "fallbackpath"):
File "edenscm/mercurial/util.py", line 190, in safehasattr
return getattr(thing, attr, _notset) is not _notset
# attr = 'fallbackpath'
File "edenscm/mercurial/util.py", line 904, in __get__
result = self.func(obj)
File "edenscm/hgext/remotefilelog/shallowrepo.py", line 42, in fallbackpath
"no remotefilelog server " "configured - is your .hg/hgrc trusted?"
Abort: no remotefilelog server configured - is your .hg/hgrc trusted?
abort: no remotefilelog server configured - is your .hg/hgrc trusted?
Fix it by making prefetchdraftparents non-fatal. This would hopefully unblock
narrow-heads rollout.
Reviewed By: xavierd
Differential Revision: D19957251
fbshipit-source-id: e65bbe6bf422776effe49055f7332ec538177a41
Summary:
Right now, if the client establishes connection to more than one peer, the
last one to proces `clienttelemetry` wireproto command gets the honor to set
`server_realhostname`. This is not desirable. Specifically, when we have
`fallbackpath` set up for remotefilelog/treemanifest and the prefetch happens
after pulling a commit, we get the hostname of the fallbackpath server,
while losing the hostname of the original `getbundle` server.
Reviewed By: DurhamG
Differential Revision: D19837570
fbshipit-source-id: fdc41565a5dfe670df3caf3b034196c4b7bdf6d9
Summary:
Assigning to `__metaclass__` only works in Python 2. In Python 3 this
attribute is ignored, and you instead need to specify `metaclass=` in your
class signature.
This adds a new `pycompat.ABC` base class to make it easier to define abstract
classes that are compatible with both Python 2 and 3.
Reviewed By: sfilipco
Differential Revision: D19672078
fbshipit-source-id: c56eb6c7266f962ef9c13c742e6ac0651b9c19c9
Summary:
Add a partial stub so pyre stops complaining about "bindings" being unknown.
Pyre's "search_path" was adjusted to read the new "pystubs" directory.
Reviewed By: DurhamG
Differential Revision: D19669131
fbshipit-source-id: a592ed411d1689058405689eb657e543b7172774
Summary:
D19657765+D19656773 made bundle2 use the standard urllib which turns
bytes into strings. This messes up bundle2 caps encoding and decoding. Caps
encoding/decoding is a nested mess of strings and bytes, so let's make it all
strings until the last minute it goes over the wire.
Reviewed By: quark-zju
Differential Revision: D19665131
fbshipit-source-id: a2fd6e309ed6da2b927d2f4ead97825a16bfcaac
Summary: We need to encode the key to be able to hash it.
Reviewed By: genevievehelsel
Differential Revision: D19651373
fbshipit-source-id: 214a3d127eb71a1c9cbf4ed1fec4f5a58f7f5195
Summary:
bundle2 parameter values are bytes, so when we compare the treemanifest
category with the bundle2 part category we need to encode the bundle2 part one.
Also fix wirepack to read utf8 paths.
Reviewed By: xavierd
Differential Revision: D19650197
fbshipit-source-id: 67da3bd8244f7953ec608019844988b20ad1929f
Summary: most instances of this `__non_zero__` ->` __bool__` were already aliased upstream, just updates some cases in which it was not.
Reviewed By: lisroach, quark-zju, sfilipco
Differential Revision: D19591210
fbshipit-source-id: 624fcfbb35b463f2660a47ca50d28d3dcd9e2cb0
Summary:
The Rust Manifest is deployed practically everywhere.
Removing the C++ code from being compiled in Mercurial along with all of the cstore extension.
Reviewed By: quark-zju
Differential Revision: D19543248
fbshipit-source-id: d632e171175e6866563c1aa0808a099b67bd937d
Summary:
The Mercurial codebase contains over 500 errors, let's ignore them for now, we
can go back to them later to fix them.
Besides the manual change to .pyre_configuration.local, the changes were
generated with:
pyre --output=json check | pyre-upgrade fixme
Reviewed By: singhsrb
Differential Revision: D18803908
fbshipit-source-id: 724db7bd864c0de47a97ef2092bdee9f2cda531f
Summary:
In preparation for merging fb-mercurial sources to the Eden repository,
move everything from the top-level directory into an `eden/scm`
subdirectory.
