Summary:
Right now, all of our manifest parsing and evaluation is in the repo() class, but this is a design mistake. Over a repo's convert lifetime, a single repo will have many different manifests, based on branch, and location in the commit history. What's worse is that the current design makes it hard to build unit tests and new features like include evaluation.
This commit creates a whole new class called repomanifest, that represents a specific manifest (and its included files). It also has unit tests to test the various operations that the manifest performs, such as path and revision mapping. This commit does not modify the existing converter code outside of the class to use this new implementation.
Reviewed By: tchebb
Differential Revision: D19402995
fbshipit-source-id: b97dadcc595c6332f4495460618317194873a780
Summary:
In the past I saw test breakages where the stderr from the remote ssh process
becomes incomplete. It's hard to reproduce by running the tests directly.
But inserting a sleep in the background stderr thread exposes it trivially:
```
# sshpeer.py:class threadedstderr
def run(self):
# type: () -> None
while not self._stop:
buf = self._stderr.readline()
+ import time
+ time.sleep(5)
if len(buf) == 0:
break
```
Example test breakage:
```
--- a/test-commitcloud-sync.t
+++ b/test-commitcloud-sync.t.err
@@ -167,8 +167,7 @@ Make a commit in the first client, and sync it
$ hg cloud sync
commitcloud: synchronizing 'server' with 'user/test/default'
backing up stack rooted at fa5d62c46fd7
remote: pushing 1 commit:
- remote: fa5d62c46fd7 commit1
commitcloud: commits synchronized
finished in * (glob)
....
```
Upon investigation it's caused by 2 factors:
- The connection pool calls pipee.close() before pipeo.close(), to workaround
an issue that I suspect solved by D19794281.
- The new threaded stderr (pipee)'s close() method does not actually closes the
pipe immediately. Instead, it limits the text to read to one more line at
most, which causes those incomplete messages.
This diff made the following changes:
- Remove the `pipee.close` workaround in connectionpool.
- Remove `pipee.close`. Embed it in `pipee.join` to prevent misuses.
- Add detailed comments in sshpeer.py for the subtle behaviors.
Reviewed By: xavierd
Differential Revision: D19872610
fbshipit-source-id: 4b61ef8f9db81c6c347ac4a634e41dec544c05d0
Summary:
This makes `peer.close()` actually close the ssh connection if it's an
sshpeer. This affects the `clone` path to actually clean up the ssh connection
so we don't depend on (fragile) `__del__`.
I traced the code back to peerrepository.close in 2011 [1]. At that time it
seems the codebase depends on `__del__`. Nowadays the codebase calls `close()`
properly so I think it's reasonable to make the change.
[1]: https://www.mercurial-scm.org/repo/hg/rev/d747774ca9da.
Reviewed By: ikostia
Differential Revision: D19911393
fbshipit-source-id: ea640d1cd82ffcb786e22f47da8116c7f50a4690
Summary:
The added function can be used by extensions to run extra logic before the
"clone" function closes the repos or peers.
This is needed to make the next diff work. Otherwise extensions like remotenames will try to write to a closed sshpeer and cause errors.
Reviewed By: DurhamG
Differential Revision: D19911390
fbshipit-source-id: ca1364e808cebb632e051fbbdcfe4bf0dca721bc
Summary: Updates the C code to support unicode filenames and states.
Reviewed By: simpkins
Differential Revision: D19786275
fbshipit-source-id: e7aeb029b792818b1b1a9c5d3028640b56522235
Summary: There is no need to open a transaction otherwise.
Reviewed By: DurhamG
Differential Revision: D20109840
fbshipit-source-id: e47adaaeea2d7565f3629701d8de4a67d4b55182
Summary:
Verifying the changelog is quite slow and we've had more users needing
to run hg recover these days. Let's finally get rid of the verify step.
Reviewed By: simpkins
Differential Revision: D20109706
fbshipit-source-id: a512d9e11716514bce986b0e3a26347fe6afd955
Summary: Most of the fixes related to encoding in `patch.py`
Reviewed By: DurhamG
Differential Revision: D19713378
fbshipit-source-id: 66ccbd0fc7826ab2d4c05173c7e9edb96700d106
Summary: As I work, it's getting harder and harder to keep my multiple changes from introducing merge conflicts between different branches. We need to break out the repo_source's implementation in to a bunch of different files to make it easier to keep things separate.
Reviewed By: zhonglowu, tchebb
Differential Revision: D20015946
fbshipit-source-id: bf954ac581e5ca9e43c091b6b1b4c539c14471f2
Summary:
`treedirstatemap._repacked` is sometimes set in write(), but does not appear
to be used anywhere. Remove it. (I noticed this since Pyre complains about
it if you enable type checking for `write()`)
Reviewed By: xavierd
Differential Revision: D19958219
fbshipit-source-id: a55e237865160191d814ed950f69c3113bec4f64
Summary:
Add type annotations for the propertycache type.
Unfortunately at the moment Pyre still can't properly type check code that
uses this class, as it does not understand the special `__get__()` method.
It looks like support for this is hopefully coming in D19206575.
Reviewed By: xavierd
Differential Revision: D19958223
fbshipit-source-id: 0f8f15fc6935ec3feaef41d3be373a85225276fe
Summary:
Add type annotations for `dirstate.status()` and
`filesystem.pendingchanges()`
Unfortunately Pyre appears to choke when processing the `dirstate.status()`
function, and currently does not actually report type errors inside this
function at the moment. I've let the Pyre team know about this.
(If Pyre did work correctly it would report one issue since it doesn't realy
understand the `rootcache` decorator applied to `dirstate._ignore`)
Reviewed By: xavierd
Differential Revision: D19958226
fbshipit-source-id: a1cd4b9402a0a449481035cee819533c56b9b336
Summary:
This module previously used to handle deciding how a particular module should
be imported if it had multiple versions (e.g., pure Python or native).
However, as of D18819680 it was changed to always import the native C version.
Lets go ahead and remove it entirely now. Using `policy.importmod` simply
makes it harder for type checkers to figure out the actual module that will be
used.
The only functionality that `policy.importmod()` still provided was verifying
that the module contained a "version" field that looked like what was
expected. In practice these version numbers are not bumped often, so this
doesn't really seem to provide much value in checking that we imported the
correct version that we expected to be shipped with this release.
Reviewed By: xavierd
Differential Revision: D19958227
fbshipit-source-id: 05f1d027d0a41cf99c4aa93cb84a51e830305077
Summary:
Add *.pyi type stub files for most of the native C extensions.
This allows Pyre to type check functions that use these extensions.
These type annotations likely aren't complete, but contain enough information
to allow Pyre to pass cleanly on the existing type-checked locations in the
code using these modules.
Reviewed By: xavierd
Differential Revision: D19958220
fbshipit-source-id: 85dc39a16e595595a174a8e59e419c418d3531be
Summary:
This moves the build rules for the extensions in mercurial/cext into a TARGETS
file in this directory.
This will allow us to start writing `*.pyi` files that contain type
information for these modules, and store them alongside the corresponding `.c`
files. By having the build rules in the top-level `eden/scm` directory we
would have needed to keep the `.pyi` files for these modules directly in the
`eden/scm` directory instead, as the namespace for the `pyi` files is assumed
to be the basemodule plus their path relative to the TARGETS file.
Reviewed By: xavierd
Differential Revision: D19958222
fbshipit-source-id: fdc26ead16663036ffa2562a96eb1649f91cba81
Summary:
The last diff fixed this for fsmonitor. Let's skip these same paths for
non-fsmonitor.
Reviewed By: quark-zju
Differential Revision: D20014808
fbshipit-source-id: 02e3cd9aa29d9c024ba3e8e42a46e21a7c8dfc30
Summary:
Watchman may report invalid utf-8 filenames, even after they've been
deleted. Let's skip them, and print a warning.
Reviewed By: sfilipco
Differential Revision: D20012187
fbshipit-source-id: b13550918a8330ef3eb5c546105d1e054dcb7724
Summary:
Whenever remotefilelog.cacheprocess2 is set, remotefilelog.cachekey is also
set, but the later is not be present when remotefilelog.cacheprocess is. Since
remotefilelog.cacheprocess already includes the cachekey, let's not add it
twice.
This also fixes the issue where hg_memcache_client would die early due to being
passed too many arguments.
Reviewed By: DurhamG
Differential Revision: D20014792
fbshipit-source-id: 8ed6775f70cf967d1c069f8acdb5a782ee819090
Summary:
This error handling can be extremely slow: calling `self.node()` can end up
triggering a linkrev scan of the changelog, which can take over 5 minutes.
If we did want to add this back in the future we would need some sort of API
on `filectx` to try and get the node ID only if it was cheap, and that would
fail fast if this is using remotefilelog and trying to get the node ID will
require scanning the changelog.
Note that KeyError can occur fairly regularly when invoked in long-lived
commands like `hg debugedenimporthelper`. If we are asked about data in a new
commit that was added since this repository was originally opened a KeyError
will be thrown here (in which case `debugedenimporthelper` will call
`repo.invalidate()` and then retry).
Reviewed By: quark-zju
Differential Revision: D20010279
fbshipit-source-id: 0e9b4c163cb9256de57daa91eed70a3736cb1075
Summary: There are two copies of pywatchman in fbcode (!) and some changes didn't make it into the edenscm copy.
Reviewed By: quark-zju
Differential Revision: D19794480
fbshipit-source-id: bcc85e0d3efc225d94b8bfa1e433f6e9cc024643
Summary:
Some of our upcoming repo merges will make it infeasible for someone to
use a full checkout. Let's add a config that will warn users of this. It has a
few levels, starting with a suppressable hint, then a non-suppressable warning,
then a suppressable exception, then a non-suppressable exception.
Reviewed By: ikostia
Differential Revision: D19974408
fbshipit-source-id: bad35a477ad8626dbc0977465368f5d71007e2d5
Summary:
On Windows, there are *two* 8-bit encodings for each process.
* The ANSI code page is used for all `...A` system calls, and this is what
Mercurial uses internally. It can be overridden using the `--encoding`
command line option.
* The OEM code page is used when outputing to the console. Mercurial has no
concept of this, and instead renders to the console using the ANSI code page,
which results in mojibake like "Θ" instead of "é".
Add the concept of an `outputencoding`. If this differs from `encoding`, we
convert from the local encoding to the output encoding before writing to the
console.
On non-Windows platforms, this defaults to the same encoding as the local encoding,
so this is a no-op unless `--outputencoding` is manually specified.
On Windows, this defaults to the codepage given by `GetOEMCP`, causing output
to be converted to the OEM codepage before being printed.
For ordinary strings, the local encoded version is wrapped by `localstr` if the
encoding does not round-trip cleanly. This means the output encoding works
even if the character is not represented in the local encoding.
Unfortunately, the templater is not localstr-clean, which means strings can get
flattened down to the local encoding and the original code points are lost. In
this case we can only output characters which are in the intersection of the
encoding and the output encoding.
Most US English Windows systems use cp1252 for the ANSI code page and cp437 for
the OEM code page. These both contain many accented characters, so users with
accented characters in their names will now see them correctly rendered.
All of this only applies to Python 2.7. In Python 3, everything is Unicode,
the `--encoding` and `--outputencoding` options do nothing, and it just works.
Reviewed By: quark-zju, ikostia
Differential Revision: D19951381
fbshipit-source-id: d5cb8b5bfe2bc131b2e6c3b892137a48b2139ca9
Summary:
`hg rage` generates the rage in the user's encoding. Since pastes are expected
to be in UTF-8, non-UTF-8 encodings result in garbled pastes.
Similarly, the lines-dec graph renderer uses escape sequences that won't work
on web pages, and the lines graph renderer uses curved lines which don't
render very well either. Force the use of the lines-square graph renderer,
which renders well.
Reviewed By: quark-zju
Differential Revision: D19951382
fbshipit-source-id: d1a5fd2ef195658f9bf10210088031474355f168
Summary:
The Rust graph renderer expects the message to be a unicode string, so ensure
we convert it from the local encoding before passing it to Rust.
Reviewed By: quark-zju
Differential Revision: D19951383
fbshipit-source-id: 644862c63873079364cb9902bd1bb49de8aa1ab9
Summary:
As of 63c471ad8a4ba0bebd1acf70569bcdcefc3fffbf in upstream Dulwich, it
now turns commands into unicode. Unfortunately, _ssh.py in hggit sees that the
type is no longer str or bytes and thinks it's an array and puts spaces between
every letter, causing it to break.
Let's allow unicode. This broke because dulwich was recently upgraded.
Reviewed By: sfilipco
Differential Revision: D19983215
fbshipit-source-id: 059756905bf4b2c73009001b078c8723ae378246
Summary: This should get rid of the extraneous uninitialized attribute errors related to `setUp` and abstract classes.
Reviewed By: simpkins
Differential Revision: D19964487
fbshipit-source-id: 52d5a6496e372d99d4398473f9ed7672228a76f5
Summary:
Update includes to the third-party xdiff.h file to use absolute includes
from the repository root. This allows many parts of our internal build
tooling to work better, including automatic dependency processing.
Reviewed By: xavierd
Differential Revision: D19958228
fbshipit-source-id: 341dd8c94f1138cf4a387b92e1817b2a286d6aa1
Summary:
Update the C files under edenscm/mercurial/cext to use absolute includes from
the repository root. Also update a few of the libraries in edenscm/mercurial
that the cext code depends on.
This makes these files easier to build with Buck in fbsource, and reduces the
number of places where we have to use deprecated Buck functionality to help
find these headers. This also allows autodeps to work with the build targets
for these rules.
Reviewed By: xavierd
Differential Revision: D19958221
fbshipit-source-id: e6e471583a795ba5773bae5f16ed582c9c5fd57e
Summary:
Remove `thirdparty/pyre2/__init__.py` from the `libhg` sources list.
We don't compile the `thirdparty/pyre2/_re2.cc` file in the fbcode build, so
importing the `__init__.py` module from this package just triggers an
ImportError when the code tries to use it. The code then always falls back to
using the version of pyre2 included from the `fb-re2` wheel.
Dropping the `__init__.py` module from our library should simply trigger an
ImportError earlier when we can't even find this file, and the code will still
fall back to using `fb-re2`.
Including this `__init__.py` file just causes issues for type checking, since
it causes us to try and type check this file even though its dependencies are
not present.
Reviewed By: xavierd
Differential Revision: D19958224
fbshipit-source-id: 34ea8806b6ee9377f17a9318c64c91ec242225df
Summary:
Some of the methods in eden_dirstate_map.py had comments that were close to
type annotations that were added a couple years ago. Update them to proper
type comments that can be recognized by Pyre and mypy.
Also remove the unused create_clone_of_internal_map() method.
Reviewed By: chadaustin, xavierd
Differential Revision: D19958225
fbshipit-source-id: b753c030acb15cf4f8d8c536614e657ee1bcba52
Summary:
Update the `eden_dirstate_map` class to store `dirstatetuple` objects instead
of plain tuples in its `_map` member variable. Without this the `filefoldmap`
code that is used on Windows fails, as it directly accesses `self._map` and
expects it to contain `dirstatetuple` objects.
Reviewed By: DurhamG, pkaush
Differential Revision: D19841881
fbshipit-source-id: ddb7523b598cfd8ec8719a8a74446cefcb411358
Summary:
Not being able to prefetch draft parent trees should not be considered as a
fatal error.
This code path is causing trouble with narrow-heads clone:
1. Streaming clone. The client gets a changelog.
2. The client runs "pull" to get new commits. The prefetchdraftparents code path runs.
3. The client has stale remote names, and public() is lagging. `prefetchdraftparents`
will try to fetch trees at the old master, but the repo is not configured properly.
That causes a stacktrace like:
$ /usr/bin/hg --config 'extensions.fsmonitor=!' clone --shallow -U --config 'ui.ssh=ssh -oControlMaster=no' --configfile /etc/mercurial/repo-specific/www.rc ssh://hg.fb.com/repo repo
connected to hg.fb.com
streaming all changes
searching for changes
adding commits
adding manifests
adding file changes
added 1 commits with 0 changes to 0 files # <<<< No traceback if this says "0 commit".
Traceback (most recent call last):
File "edenscm/hgext/remotenames.py", line 1464, in exclonecmd
orig(ui, *args, **opts)
File "edenscm/hgext/remotefilelog/__init__.py", line 433, in cloneshallow
orig(ui, repo, *args, **opts)
File "edenscm/mercurial/commands/__init__.py", line 1615, in clone
shareopts=shareopts,
# shareopts = {'mode': 'identity'}
File "edenscm/mercurial/hg.py", line 741, in clone
exchange.pull(local, srcpeer, revs, streamclonerequested=stream)
File "edenscm/mercurial/util.py", line 621, in __exit__
self.close()
File "edenscm/mercurial/transaction.py", line 46, in _active
return func(self, *args, **kwds)
File "edenscm/mercurial/transaction.py", line 543, in close
self._postclosecallback[cat](self)
# cat = bin('6472616674706172656e74747265656665746368')
File "edenscm/hgext/treemanifest/__init__.py", line 490, in _parenttreefetch
self.prefetchtrees([c.manifestnode() for c in draftparents])
# c = <changectx b5ad643b3009>
# draftparents = [<changectx b5ad643b3009>]
File "edenscm/hgext/treemanifest/__init__.py", line 522, in prefetchtrees
self._prefetchtrees("", mfnodes, basemfnodes, [], depth)
# basemfnodes = [bin('a25f17018d7cd07f1f6bc3076f95c5980ba087a9')]
# mfnodes = [bin('ad717aac7700e783a1d84f3330d13a7731a4726a')]
File "edenscm/hgext/treemanifest/__init__.py", line 529, in _prefetchtrees
fallbackpath = getfallbackpath(self)
File "edenscm/hgext/treemanifest/__init__.py", line 2173, in getfallbackpath
if util.safehasattr(repo, "fallbackpath"):
File "edenscm/mercurial/util.py", line 190, in safehasattr
return getattr(thing, attr, _notset) is not _notset
# attr = 'fallbackpath'
File "edenscm/mercurial/util.py", line 904, in __get__
result = self.func(obj)
File "edenscm/hgext/remotefilelog/shallowrepo.py", line 42, in fallbackpath
"no remotefilelog server " "configured - is your .hg/hgrc trusted?"
Abort: no remotefilelog server configured - is your .hg/hgrc trusted?
abort: no remotefilelog server configured - is your .hg/hgrc trusted?
Fix it by making prefetchdraftparents non-fatal. This would hopefully unblock
narrow-heads rollout.
Reviewed By: xavierd
Differential Revision: D19957251
fbshipit-source-id: e65bbe6bf422776effe49055f7332ec538177a41
Summary:
This will allow us to improve our dashboards filtering out errors we are
responsible for, like missing certs on the machines.
Reviewed By: mitrandir77
Differential Revision: D19950614
fbshipit-source-id: 73503e984dfe8513a700fdcb2fc36b1618c20a4f
Summary:
Commit messages and extras can be unbounded in size. This can cause problems if users create commits with exceptionally large messages or extras. Mercurial will commit these to the changelog, increasing its size. On Mononoke, large commit messages may go over the cacheing threshold, resulting in poor performance for requests involving these commits as Mononoke will need to reload on every access.
Commit messages should not usually be that large. Mostly likely it will happen by accident, e.g. through use of `hg commit -l some-large-file`. Prevent this from happening by accident by adding configuration for soft limits when creating commits.
If a user really does need to create a commit with a very large message or extras, they can override using the config option.
Reviewed By: xavierd
Differential Revision: D19942522
fbshipit-source-id: 09b9fe1f470467237acc1b20286d2b1d2ab25613
Summary:
This parameter was originally removed in D12811551, but re-added in D12855935
due to the fact that at the time the `eden_dirstate.py` and `dirstate.py`
files were deployed in separate RPMs and could not be updated together
atomically. We now deploy these files together, so we can drop this extra
unnecessary argument.
Reviewed By: chadaustin
Differential Revision: D19913057
fbshipit-source-id: 0f0b4fde4b3124a8fc5bb568551b4e67de14d410
Summary:
Sometimes the treestate points to an unknown commit (ex. aborted transaction
might strip commits). While `debugrebuilddirstate -r HASH --hidden` is able to
fix it, it is too slow.
This diff adds treestate repair logic to the `doctor` command. It scans through
the treestate files, find a most recent `Root` entry with `p1` pointing to a
known commit.
This can be much faster than `debugrebuilddirstate` in some cases, because the
watchman clock might still be valid, and the NEED_CHECK file list might still
be small. In that case, `status` can still be fast.
Since treestate atomically updates all information needed for `status`
calculation (parents, need-check-files (or, "non-normal files"), watchman-clock
(only with fsmonitor), and stat for clean files). Reverting to a previous state
is still atomic. Correctness-wise, this is equivalent to aborting a "large"
transaction, and restoring treestate data to the state before the transaction.
It should be consistent, and the next `status` call won't mis-report files like
the dangerous `debugsetparents` command.
Reviewed By: DurhamG
Differential Revision: D19864422
fbshipit-source-id: d5d2f8b43a0c15ea2ac0e3c164edec7deeb8451f
Summary:
See the test change. Without this change repairing the changelog won't give the
user back a working repo.
Reviewed By: markbt
Differential Revision: D19864421
fbshipit-source-id: b84582c5302469828c8cfcb3db362ea82f2eea63
Summary:
Reuse utilities in the fixcorrupt extension to repair changelog.
This is better than fixcorrupt because `hg doctor` does not require a repo
object. Some messages are updated so they become more consistent with the
rest of `hg doctor`.
The main motivation is to get changelog fixed early, so other repair logic can
check if a commit hash is known by changelog or not.
Reviewed By: markbt
Differential Revision: D19864418
fbshipit-source-id: 6f95c6c6191d7db2a474a07a5278a857cf41d8e2
Summary:
Run 'edenfsctl doctor' on an edenfs repo. If there is no current repo, it might
be caused by edenfs daemon stopped running. So let's also run edenfsctl doctor
in that case.
Reviewed By: markbt
Differential Revision: D19864419
fbshipit-source-id: d2a49a126a040845b88b4883d214162326d08d8d
Summary:
We're seeing a user have issues because their username contains unicode
characters and sampling's use of json doesn't handle it well. I've not been able
to repro it unfortunately, but let's go ahead and switch sampling to use
mercurial.json.
Differential Revision: D19895419
fbshipit-source-id: a1f087d1e2c7568488c2b8d54f267bd5c8266202
Summary: There's still some issues, but it's a lot closer.
Reviewed By: quark-zju
Differential Revision: D19802023
fbshipit-source-id: da539094cbc0ba3542e4b5fd3d49f5f80455ec23
