facebook/sapling
1
1
Fork 0
mirror of https://github.com/facebook/sapling.git synced 2024-10-11 01:07:15 +03:00
Code Issues Projects Releases Wiki Activity
55,995 Commits 25 Branches 62 Tags 325 MiB
b6cea95ea5
Commit Graph

955 Commits

Author SHA1 Message Date
Adam Simpkins
375df3b2c0 add type annotations for dirstate.status() 
Summary:
Add type annotations for `dirstate.status()` and
`filesystem.pendingchanges()`

Unfortunately Pyre appears to choke when processing the `dirstate.status()`
function, and currently does not actually report type errors inside this
function at the moment.  I've let the Pyre team know about this.
(If Pyre did work correctly it would report one issue since it doesn't realy
understand the `rootcache` decorator applied to `dirstate._ignore`)

Reviewed By: xavierd

Differential Revision: D19958226

fbshipit-source-id: a1cd4b9402a0a449481035cee819533c56b9b336
 2020-02-21 13:54:20 -08:00
Adam Simpkins
0bc787c4c0 remove the mercurial.policy module 
Summary:
This module previously used to handle deciding how a particular module should
be imported if it had multiple versions (e.g., pure Python or native).
However, as of D18819680 it was changed to always import the native C version.

Lets go ahead and remove it entirely now.  Using `policy.importmod` simply
makes it harder for type checkers to figure out the actual module that will be
used.

The only functionality that `policy.importmod()` still provided was verifying
that the module contained a "version" field that looked like what was
expected.  In practice these version numbers are not bumped often, so this
doesn't really seem to provide much value in checking that we imported the
correct version that we expected to be shipped with this release.

Reviewed By: xavierd

Differential Revision: D19958227

fbshipit-source-id: 05f1d027d0a41cf99c4aa93cb84a51e830305077
 2020-02-21 13:54:19 -08:00
Adam Simpkins
fabc6d628f start adding type annotations for the cext code 
Summary:
Add *.pyi type stub files for most of the native C extensions.
This allows Pyre to type check functions that use these extensions.

These type annotations likely aren't complete, but contain enough information
to allow Pyre to pass cleanly on the existing type-checked locations in the
code using these modules.

Reviewed By: xavierd

Differential Revision: D19958220

fbshipit-source-id: 85dc39a16e595595a174a8e59e419c418d3531be
 2020-02-21 13:54:19 -08:00
Adam Simpkins
446ebff30f move mercurial/cext TARGETS definitions into their subdirectory 
Summary:
This moves the build rules for the extensions in mercurial/cext into a TARGETS
file in this directory.

This will allow us to start writing `*.pyi` files that contain type
information for these modules, and store them alongside the corresponding `.c`
files.  By having the build rules in the top-level `eden/scm` directory we
would have needed to keep the `.pyi` files for these modules directly in the
`eden/scm` directory instead, as the namespace for the `pyi` files is assumed
to be the basemodule plus their path relative to the TARGETS file.

Reviewed By: xavierd

Differential Revision: D19958222

fbshipit-source-id: fdc26ead16663036ffa2562a96eb1649f91cba81
 2020-02-21 13:54:18 -08:00
Liubov Dmitrieva
93bb52a3b2 unify bad certs warnings/errors 
Summary: Better engineering: let's finally unify those warnings

Reviewed By: quark-zju

Differential Revision: D20029852

fbshipit-source-id: b6522b7384e763650f96a482cb22a12935a680f7
 2020-02-21 12:05:19 -08:00
Durham Goode
fc12041053 unicode: fix non-fsmonitor encountering non-utf8 paths on disk 
Summary:
The last diff fixed this for fsmonitor. Let's skip these same paths for
non-fsmonitor.

Reviewed By: quark-zju

Differential Revision: D20014808

fbshipit-source-id: 02e3cd9aa29d9c024ba3e8e42a46e21a7c8dfc30
 2020-02-21 11:52:41 -08:00
Durham Goode
fa31ef23e0 utf8: skip invalid utf-8 filenames from watchman 
Summary:
Watchman may report invalid utf-8 filenames, even after they've been
deleted. Let's skip them, and print a warning.

Reviewed By: sfilipco

Differential Revision: D20012187

fbshipit-source-id: b13550918a8330ef3eb5c546105d1e054dcb7724
 2020-02-21 08:05:08 -08:00
Durham Goode
78ada1f780 utf8: use cpython_ext::Str for error strings 
Summary:
Error strings were being converted to unicode if they contained certain
characters. This caused python 2 Mercurial to throw various errors when it tried
to turn them into strings to report errors.

Let's return cpython_ext::Str instead of String.

Reviewed By: sfilipco

Differential Revision: D20012188

fbshipit-source-id: af6fa7d98d68e3c188292e4972cfc1bdb758dbdf
 2020-02-21 08:05:08 -08:00
Xavier Deguillard
d1715ef91a remotefilelog: only append remotefilelog.cachekey to cacheprocess2 
Summary:
Whenever remotefilelog.cacheprocess2 is set, remotefilelog.cachekey is also
set, but the later is not be present when remotefilelog.cacheprocess is. Since
remotefilelog.cacheprocess already includes the cachekey, let's not add it
twice.

This also fixes the issue where hg_memcache_client would die early due to being
passed too many arguments.

Reviewed By: DurhamG

Differential Revision: D20014792

fbshipit-source-id: 8ed6775f70cf967d1c069f8acdb5a782ee819090
 2020-02-20 19:32:43 -08:00
Adam Simpkins
aea44066d4 remove verbose KeyError handling in filectx.data() 
Summary:
This error handling can be extremely slow: calling `self.node()` can end up
triggering a linkrev scan of the changelog, which can take over 5 minutes.

If we did want to add this back in the future we would need some sort of API
on `filectx` to try and get the node ID only if it was cheap, and that would
fail fast if this is using remotefilelog and trying to get the node ID will
require scanning the changelog.

Note that KeyError can occur fairly regularly when invoked in long-lived
commands like `hg debugedenimporthelper`.  If we are asked about data in a new
commit that was added since this repository was originally opened a KeyError
will be thrown here (in which case `debugedenimporthelper` will call
`repo.invalidate()` and then retry).

Reviewed By: quark-zju

Differential Revision: D20010279

fbshipit-source-id: 0e9b4c163cb9256de57daa91eed70a3736cb1075
 2020-02-20 17:43:48 -08:00
Chad Austin
3748788c03 sync bser fixes from pywatchman 
Summary: There are two copies of pywatchman in fbcode (!) and some changes didn't make it into the edenscm copy.

Reviewed By: quark-zju

Differential Revision: D19794480

fbshipit-source-id: bcc85e0d3efc225d94b8bfa1e433f6e9cc024643
 2020-02-20 14:54:29 -08:00
Xavier Deguillard
44c4f2f5d9 revisionstore: add copyfrom information to the LFS pointer 
Summary:
Mercurial filenode hash is computed by including the copy information in the
blob header. Before computing the blob content hash, or returning it to the
upper layers, we need to either strip or reconstruct this header appropriately.

Reviewed By: DurhamG

Differential Revision: D19975887

fbshipit-source-id: 7555e7219e50f4d18ec677fdecc216ee705d7af4
 2020-02-20 14:28:52 -08:00
Xavier Deguillard
7fb75ce4f0 lfs: move contenthash computation to the enum impl 
Summary: This will make it easier to support more hash schemes in the future.

Reviewed By: DurhamG

Differential Revision: D19975888

fbshipit-source-id: 8b8ce3b20d72199bac3cd20a48475b5ab56bfc52
 2020-02-20 14:28:52 -08:00
Xavier Deguillard
cd56a8b39a revisionstore: move Arc outside of the stores 
Summary:
With the Arc embedded into the store themselves, this forces a second
allocation in order to use them as trait objects. Since in most cases, we do
not want the stores themselves to be cloneable, we can move the Arc outside and
thus reduce the number of pointer indirection.

Reviewed By: DurhamG

Differential Revision: D19867568

fbshipit-source-id: 9cd126831fe2b9ee715472ac3299b7a09df95fce
 2020-02-20 14:28:52 -08:00
Xavier Deguillard
7c1a623d8a revisionstore: add the LfsStore to the ContentStore 
Summary:
The ContentStore now can read LFS blobs from both the shared cache, and the
local store.

Reviewed By: DurhamG

Differential Revision: D19866249

fbshipit-source-id: a6fb3523495e9d3832613b56438f631cfa552b91
 2020-02-20 14:28:51 -08:00
Xavier Deguillard
58d9d92e88 revisionstore: simplify ContentStore/MetadataStore initialization a bit 
Summary:
With the LFS store being added, and the indexedlog being soon used for trees,
this simplification should help in formalizing the hierarchy of files/folders.

It will look like the following:
  <root dir>/lfs: for the lfs store
  <root dir>/indexedlog*: for the indexedlog
  <root dir>/foobar: for a hypothetical foobar store

For manifests, <root dir> will therefore be: <store dir>/manifests. The
unfortunate part is that the current tree data lives under
<store dir>/packs/manifests. As packfiles will be replaced, this small
discrepency is acceptable.

Reviewed By: DurhamG

Differential Revision: D19866248

fbshipit-source-id: 7ef59ef7df19149b19a529b4f4a45a479cc9d23b
 2020-02-20 14:28:51 -08:00
Xavier Deguillard
f512b5658d revisionstore: add an LfsStore 
Summary:
This is the first step in having a stronger integration between LFS blobs and
the ContentStore abstraction. The 2 main difference between the Python based
LFS implementation and this one are:
 - pointers are not stored alongside plain data,
 - blobs are split between local and shared blobs

As of now, no reclamation is being performed for shared blobs, blobs aren't
fetched or uploaded. This will come in future diffs.

Reviewed By: DurhamG

Differential Revision: D19859291

fbshipit-source-id: 45000fc574e6fbd6d3487f4966cad4f49dab731c
 2020-02-20 14:28:51 -08:00
Durham Goode
51720717d8 sparse: add config for warning for full checkouts 
Summary:
Some of our upcoming repo merges will make it infeasible for someone to
use a full checkout. Let's add a config that will warn users of this. It has a
few levels, starting with a suppressable hint, then a non-suppressable warning,
then a suppressable exception, then a non-suppressable exception.

Reviewed By: ikostia

Differential Revision: D19974408

fbshipit-source-id: bad35a477ad8626dbc0977465368f5d71007e2d5
 2020-02-20 10:58:53 -08:00
Mark Thomas
5168c29e12 encoding: use correct output encoding on windows 
Summary:
On Windows, there are *two* 8-bit encodings for each process.

* The ANSI code page is used for all `...A` system calls, and this is what
  Mercurial uses internally.  It can be overridden using the `--encoding`
  command line option.

* The OEM code page is used when outputing to the console.  Mercurial has no
  concept of this, and instead renders to the console using the ANSI code page,
  which results in mojibake like "Θ" instead of "é".

Add the concept of an `outputencoding`.  If this differs from `encoding`, we
convert from the local encoding to the output encoding before writing to the
console.

On non-Windows platforms, this defaults to the same encoding as the local encoding,
so this is a no-op unless `--outputencoding` is manually specified.

On Windows, this defaults to the codepage given by `GetOEMCP`, causing output
to be converted to the OEM codepage before being printed.

For ordinary strings, the local encoded version is wrapped by `localstr` if the
encoding does not round-trip cleanly.  This means the output encoding works
even if the character is not represented in the local encoding.

Unfortunately, the templater is not localstr-clean, which means strings can get
flattened down to the local encoding and the original code points are lost.  In
this case we can only output characters which are in the intersection of the
encoding and the output encoding.

Most US English Windows systems use cp1252 for the ANSI code page and cp437 for
the OEM code page.  These both contain many accented characters, so users with
accented characters in their names will now see them correctly rendered.

All of this only applies to Python 2.7.  In Python 3, everything is Unicode,
the `--encoding` and `--outputencoding` options do nothing, and it just works.

Reviewed By: quark-zju, ikostia

Differential Revision: D19951381

fbshipit-source-id: d5cb8b5bfe2bc131b2e6c3b892137a48b2139ca9
 2020-02-20 04:28:48 -08:00
Mark Thomas
c4acebfca1 rage: force use of utf-8 and lines-square graph renderer 
Summary:
`hg rage` generates the rage in the user's encoding.  Since pastes are expected
to be in UTF-8, non-UTF-8 encodings result in garbled pastes.

Similarly, the lines-dec graph renderer uses escape sequences that won't work
on web pages, and the lines graph renderer uses curved lines which don't
render very well either.  Force the use of the lines-square graph renderer,
which renders well.

Reviewed By: quark-zju

Differential Revision: D19951382

fbshipit-source-id: d1a5fd2ef195658f9bf10210088031474355f168
 2020-02-20 04:28:48 -08:00
Mark Thomas
1d70594590 renderdag: provide graph message in unicode 
Summary:
The Rust graph renderer expects the message to be a unicode string, so ensure
we convert it from the local encoding before passing it to Rust.

Reviewed By: quark-zju

Differential Revision: D19951383

fbshipit-source-id: 644862c63873079364cb9902bd1bb49de8aa1ab9
 2020-02-20 04:28:47 -08:00
Thomas Orozco
de9b3cdc83 run-tests: encode diff in utf-8 
Summary:
See later in this stack for motivation. This seems to work fine, and it allows
characters that don't fit latin1 when rendering diffs.

Reviewed By: markbt

Differential Revision: D19969743

fbshipit-source-id: 79c4afce5a19822d9b075d23ff4c88aa76ce2f42
 2020-02-20 02:49:38 -08:00
Jun Wu
1fead58252 cmdutil: remove allcmds from UnknownCommand 
Summary:
This makes the backtrace much shorter.

Before:

	Traceback (most recent call last):
	  File "edenscm/mercurial/cmdutil.py", line 876, in findsubcmd
	    aliases, entry = findcmd(cmd, table)
	    # cmd = 'foo'
	  File "edenscm/mercurial/cmdutil.py", line 871, in findcmd
	    raise error.UnknownCommand(cmd, allcmds)
	    # allcmds = ['debuggentrees', 'debugsuccessorssets', 'branch', 'br', 'bra', 'bran', 'branc', 'show', ...]
	    # cmd = 'foo'
	UnknownCommand: ('foo', ['debuggentrees', 'debugsuccessorssets', 'branch', 'br', 'bra', 'bran', 'branc', 'show', 'rename', 'move', 'mv', 'ren', 'rena', 'renam', 'mo', 'mov', 'commit', 'ci', 'com', 'comm', 'commi', 'debugremotebookmark', 'gc', 'debugindexedlogdatastore', 'debugexistingcasecollisions', 'debugshell', 'dbsh', 'debugsh', 'debugpython', 'debugpy', 'debugfileset', 'debugdata', 'debugundohistory', 'debugprogress', 'config', 'showconfig', 'debugconfig', 'conf', 'confi', 'paths', 'path', 'bisect', 'bi', 'bis', 'bise', 'bisec', 'debugcheckcasecollisions', 'add', 'debugrename', 'annotate', 'blame', 'blam', 'an', 'ann', 'anno', 'annot', 'annota', 'annotat', 'journal', 'j', 'jo', 'jou', 'jour', 'journ', 'journa', 'parents', 'par', 'pare', 'paren', 'parent', 'rollback', 'debugmergestate', 'identify', 'id', 'ide', 'iden', 'ident', 'identi', 'identif', 'debugformat', 'bookmarks', 'bookmark', 'bo', 'boo', 'book', 'bookm', 'bookma', 'bookmar', 'debugrevspec', 'debugwalk', 'amend', 'am', 'ame', 'amen', 'debughttp', 'hide', 'strip', 'hid', 'str', 'stri', 'tip', 'ti', 'debugcommitmessage', 'hint', 'hin', 'locate', 'loc', 'loca', 'locat', 'debugfilerevision', 'debugfilerev', 'incoming', 'in', 'debugdag', 'next', 'n', 'ne', 'nex', 'unshare', 'rebase', 'unhide', 'unh', 'unhi', 'unhid', 'grep', 'gre', 'archive', 'ar', 'arc', 'arch', 'archi', 'archiv', 'debugmarklanded', 'debugpushkey', 'debugdatapack', 'debugtreestate', 'debugtreedirstate', 'debugtree', 'debuglocks', 'debuglock', 'revert', 'rev', 'reve', 'rever', 'export', 'e', 'ex', 'exp', 'expo', 'expor', 'repack', 'shelve', 'she', 'shel', 'shelv', 'log', 'history', 'debugwaitbackup', 'debugnamecomplete', 'diff', 'd', 'di', 'dif', 'debugdrawdag', 'debugcheckoutidentifier', 'debugdate', 'debuggethistory', 'debugssl', 'init', 'ini', 'debuggetbundle', 'summary', 'sum', 'su', 'summ', 'summa', 'summar', 'unbundle', 'unb', 'unbu', 'unbun', 'unbund', 'unbundl', 'debugwaitonrepack', 'backfilltree', 'debugcrdump', 'outgoing', 'out', 'debugupgraderepo', 'debugmovescratchbookmark', 'debugrebuilddirstate', 'debugrebuildstate', 'debugancestor', 'remove', 'rm', 'rem', 'remo', 'remov', 'debugfillinfinitepushmetadata', 'sparse', 'tags', 'debugcolor', 'debugrunshell', 'debugpathcomplete', 'purge', 'clean', 'undo', 'debugindexedlog-dump', 'debugcheckstate', 'metaedit', 'met', 'meta', 'metae', 'metaed', 'metaedi', 'debugcommands', 'convert', 'debugmutationfromobsmarkers', 'graft', 'gra', 'graf', 'root', 'uncommit', 'unc', 'unco', 'uncom', 'uncomm', 'uncommi', 'verifyremotefilelog', 'copy', 'cp', 'cop', 'debugpreviewbindag', 'files', 'fi', 'fil', 'file', 'debugprocesstree', 'debugproc', 'debugdeltachain', 'fold', 'squash', 'debughistorypack', 'debug-args', 'debugcapabilities', 'debugindexedloghistorystore', 'debuggetfiles', 'debuggetfile', 'debugapplystreamclonebundle', 'redo', 'cloud', 'debugremotefilelog', 'debugsetparents', 'debugmetalog', 'debugindexedlog-repair', 'debugbindag', 'version', 'vers', 'versi', 'versio', 'status', 'st', 'sta', 'stat', 'statu', 'debugmutation', 'debugignore', 'debugobsolete', 'debugupdatecaches', 'debugwaitonprefetch', 'merge', 'mer', 'merg', 'addremove', 'addr', 'addre', 'addrem', 'addremo', 'addremov', 'share', 'debugmanifestdirs', 'serve', 'fastannotate', 'fastblame', 'fa', 'debugdiscovery', 'debugwireargs', 'push', 'debugpickmergetool', 'debugstrip', 'debugbuildannotatecache', 'isbackedup', 'reset', 'debugcomplete', 'debugfsinfo', 'debugfs', 'prune', 'rage', 'unamend', 'una', 'unam', 'uname', 'unamen', 'previous', 'prev', 'previ', 'previo', 'previou', 'continue', 'cont', 'debugsvnlog', 'debugvisibility', 'cat', 'record', 'debugrevlog', 'unshelve', 'unshe', 'unshel', 'unshelv', 'debugrebuildfncache', 'debugdirs', 'import', 'patch', 'im', 'imp', 'impo', 'impor', 'patc', 'backfillmanifestrevlog', 'debuglabelcomplete', 'split', 'spl', 'spli', 'smartlog', 'sl', 'slog', 'sm', 'sma', 'smar', 'smart', 'smartl', 'smartlo', 'debuggettrees', 'debugedenrunpostupdatehook', 'debugstatus', 'manifest', 'mani', 'tag', 'bundle', 'bu', 'bun', 'bund', 'bundl', 'debugtemplate', 'phase', 'ph', 'pha', 'phas', 'dump-trace', 'debugstore', 'rebase', 'reb', 'reba', 'rebas', 'recover', 'debugrefreshwatchmanclock', 'doctor', 'verify', 'forget', 'for', 'forg', 'forge', 'debugextensions', 'pushbackup', 'absorb', 'ab', 'abs', 'abso', 'absor', 'sf', 'debuginstall', 'histgrep', 'debugreadauthforuri', 'prefetch', 'fs', 'blackbox', 'clone', 'histedit', 'histe', 'histed', 'histedi', 'debugpvec', 'resolve', 'reso', 'resol', 'resolv', 'debugcreatestreamclonebundle', 'debugserialgetfiles', 'debugbuilddag', 'debugindex', 'debugcreatescratchbookmark', 'heads', 'hea', 'head', 'debugedenimporthelper', 'githelp', 'git', 'pull', 'pul', 'chistedit', 'debugindexdot', 'debugdirstate', 'debugstate', 'debugknown', 'backout', 'backo', 'backou', 'update', 'up', 'checkout', 'co', 'upd', 'upda', 'updat', 'che', 'chec', 'check', 'checko', 'checkou', 'help', 'hel', 'debugbundle'])
	unknown command 'foo'
	(use 'hg help' to get help)

After:

	Traceback (most recent call last):
	  File "edenscm/mercurial/cmdutil.py", line 876, in findsubcmd
	    aliases, entry = findcmd(cmd, table)
	    # cmd = 'foo'
	  File "edenscm/mercurial/cmdutil.py", line 871, in findcmd
	    raise error.UnknownCommand(cmd)
	    # cmd = 'foo'
	UnknownCommand: foo
	unknown command 'foo'
	(use 'hg help' to get help)

Reviewed By: xavierd

Differential Revision: D19959205

fbshipit-source-id: e3fa6de8edfec7510ecd29defe538dddf8b563a2
 2020-02-19 20:12:57 -08:00
Durham Goode
e25f679318 hggit: fix path handling 
Summary:
As of 63c471ad8a4ba0bebd1acf70569bcdcefc3fffbf in upstream Dulwich, it
now turns commands into unicode.  Unfortunately, _ssh.py in hggit sees that the
type is no longer str or bytes and thinks it's an array and puts spaces between
every letter, causing it to break.

Let's allow unicode. This broke because dulwich was recently upgraded.

Reviewed By: sfilipco

Differential Revision: D19983215

fbshipit-source-id: 059756905bf4b2c73009001b078c8723ae378246
 2020-02-19 17:46:33 -08:00
Jia Chen
671ca89de0 Upgrade Pyre version for eden to 2927613de6d20ee2d66e98124f3834812475e122 
Summary: This should get rid of the extraneous uninitialized attribute errors related to `setUp` and abstract classes.

Reviewed By: simpkins

Differential Revision: D19964487

fbshipit-source-id: 52d5a6496e372d99d4398473f9ed7672228a76f5
 2020-02-19 15:05:25 -08:00
Jun Wu
26e4090522 distutils_rust: retry mt.exe properly 
Summary:
This is a revised version of D19887220.

D19887220 has 2 problems:
- It can silently ignore the mt.exe error after failures of all retries.
- There is another place that `mt.exe` runs that is not covered by retry.

This diff fixes them by wrapping the `set_long_paths_manifest` function
directly so it covers two `mt.exe` places, and makes sure all retry failure
is still a failure.

Reviewed By: sfilipco

Differential Revision: D19977802

fbshipit-source-id: 774d0c42b247a7e111841cd69f71760a5544d685
 2020-02-19 13:22:39 -08:00
Adam Simpkins
6f824649ff use absolute includes to the third-party xdiff code 
Summary:
Update includes to the third-party xdiff.h file to use absolute includes
from the repository root.  This allows many parts of our internal build
tooling to work better, including automatic dependency processing.

Reviewed By: xavierd

Differential Revision: D19958228

fbshipit-source-id: 341dd8c94f1138cf4a387b92e1817b2a286d6aa1
 2020-02-19 13:05:06 -08:00
Adam Simpkins
5ffa268af2 use absolute includes for the native cext modules 
Summary:
Update the C files under edenscm/mercurial/cext to use absolute includes from
the repository root.  Also update a few of the libraries in edenscm/mercurial
that the cext code depends on.

This makes these files easier to build with Buck in fbsource, and reduces the
number of places where we have to use deprecated Buck functionality to help
find these headers.  This also allows autodeps to work with the build targets
for these rules.

Reviewed By: xavierd

Differential Revision: D19958221

fbshipit-source-id: e6e471583a795ba5773bae5f16ed582c9c5fd57e
 2020-02-19 13:05:06 -08:00
Adam Simpkins
ab8071fa35 stop including thirdparty/pyre2 in the fbcode build 
Summary:
Remove `thirdparty/pyre2/__init__.py` from the `libhg` sources list.

We don't compile the `thirdparty/pyre2/_re2.cc` file in the fbcode build, so
importing the `__init__.py` module from this package just triggers an
ImportError when the code tries to use it.  The code then always falls back to
using the version of pyre2 included from the `fb-re2` wheel.

Dropping the `__init__.py` module from our library should simply trigger an
ImportError earlier when we can't even find this file, and the code will still
fall back to using `fb-re2`.

Including this `__init__.py` file just causes issues for type checking, since
it causes us to try and type check this file even though its dependencies are
not present.

Reviewed By: xavierd

Differential Revision: D19958224

fbshipit-source-id: 34ea8806b6ee9377f17a9318c64c91ec242225df
 2020-02-19 13:05:06 -08:00
Adam Simpkins
d736cca388 fix type annotations in eden_dirstate_map.py 
Summary:
Some of the methods in eden_dirstate_map.py had comments that were close to
type annotations that were added a couple years ago.  Update them to proper
type comments that can be recognized by Pyre and mypy.

Also remove the unused create_clone_of_internal_map() method.

Reviewed By: chadaustin, xavierd

Differential Revision: D19958225

fbshipit-source-id: b753c030acb15cf4f8d8c536614e657ee1bcba52
 2020-02-19 13:05:05 -08:00
Adam Simpkins
a673c6bd4d update eden_dirstate_map to store dirstatetuple objects 
Summary:
Update the `eden_dirstate_map` class to store `dirstatetuple` objects instead
of plain tuples in its `_map` member variable.  Without this the `filefoldmap`
code that is used on Windows fails, as it directly accesses `self._map` and
expects it to contain `dirstatetuple` objects.

Reviewed By: DurhamG, pkaush

Differential Revision: D19841881

fbshipit-source-id: ddb7523b598cfd8ec8719a8a74446cefcb411358
 2020-02-19 13:05:05 -08:00
Lukas Piatkowski
c4f0887fc2 eden/scm: cover xdiff with autocargo 
Summary: Generate the Cargo.toml files inside xdiff with autocargo. This will enable Mononoke to depend on this code easily without sacrificing anything on eden/scm side.

Reviewed By: aslpavel

Differential Revision: D19948741

fbshipit-source-id: 905ff3d64b90830e5f075e4c6ed2b3de959e3f00
 2020-02-19 05:15:17 -08:00
Jun Wu
3a5b53134e hint: update revnum deprecation message 
Summary: It's 2020 now.

Reviewed By: xavierd

Differential Revision: D19958630

fbshipit-source-id: 143d57e060acb150461151b31d82bb6bfefe3c91
 2020-02-18 18:06:26 -08:00
Jun Wu
ad6eb1928b treemanifest: make prefetchdraftparents non-fatal 
Summary:
Not being able to prefetch draft parent trees should not be considered as a
fatal error.

This code path is causing trouble with narrow-heads clone:

1. Streaming clone. The client gets a changelog.
2. The client runs "pull" to get new commits. The prefetchdraftparents code path runs.
3. The client has stale remote names, and public() is lagging. `prefetchdraftparents`
   will try to fetch trees at the old master, but the repo is not configured properly.

That causes a stacktrace like:

  $ /usr/bin/hg --config 'extensions.fsmonitor=!' clone --shallow -U --config 'ui.ssh=ssh -oControlMaster=no' --configfile /etc/mercurial/repo-specific/www.rc ssh://hg.fb.com/repo repo
  connected to hg.fb.com
  streaming all changes
  searching for changes
  adding commits
  adding manifests
  adding file changes
  added 1 commits with 0 changes to 0 files  # <<<< No traceback if this says "0 commit".
  Traceback (most recent call last):
    File "edenscm/hgext/remotenames.py", line 1464, in exclonecmd
      orig(ui, *args, **opts)
    File "edenscm/hgext/remotefilelog/__init__.py", line 433, in cloneshallow
      orig(ui, repo, *args, **opts)
    File "edenscm/mercurial/commands/__init__.py", line 1615, in clone
      shareopts=shareopts,
      # shareopts = {'mode': 'identity'}
    File "edenscm/mercurial/hg.py", line 741, in clone
      exchange.pull(local, srcpeer, revs, streamclonerequested=stream)
    File "edenscm/mercurial/util.py", line 621, in __exit__
      self.close()
    File "edenscm/mercurial/transaction.py", line 46, in _active
      return func(self, *args, **kwds)
    File "edenscm/mercurial/transaction.py", line 543, in close
      self._postclosecallback[cat](self)
      # cat = bin('6472616674706172656e74747265656665746368')
    File "edenscm/hgext/treemanifest/__init__.py", line 490, in _parenttreefetch
      self.prefetchtrees([c.manifestnode() for c in draftparents])
      # c = <changectx b5ad643b3009>
      # draftparents = [<changectx b5ad643b3009>]
    File "edenscm/hgext/treemanifest/__init__.py", line 522, in prefetchtrees
      self._prefetchtrees("", mfnodes, basemfnodes, [], depth)
      # basemfnodes = [bin('a25f17018d7cd07f1f6bc3076f95c5980ba087a9')]
      # mfnodes = [bin('ad717aac7700e783a1d84f3330d13a7731a4726a')]
    File "edenscm/hgext/treemanifest/__init__.py", line 529, in _prefetchtrees
      fallbackpath = getfallbackpath(self)
    File "edenscm/hgext/treemanifest/__init__.py", line 2173, in getfallbackpath
      if util.safehasattr(repo, "fallbackpath"):
    File "edenscm/mercurial/util.py", line 190, in safehasattr
      return getattr(thing, attr, _notset) is not _notset
      # attr = 'fallbackpath'
    File "edenscm/mercurial/util.py", line 904, in __get__
      result = self.func(obj)
    File "edenscm/hgext/remotefilelog/shallowrepo.py", line 42, in fallbackpath
      "no remotefilelog server " "configured - is your .hg/hgrc trusted?"
  Abort: no remotefilelog server configured - is your .hg/hgrc trusted?
  abort: no remotefilelog server configured - is your .hg/hgrc trusted?

Fix it by making prefetchdraftparents non-fatal. This would hopefully unblock
narrow-heads rollout.

Reviewed By: xavierd

Differential Revision: D19957251

fbshipit-source-id: e65bbe6bf422776effe49055f7332ec538177a41
 2020-02-18 17:36:54 -08:00
Liubov Dmitrieva
3163d20016 commitcloud: log errors to scuba 
Summary:
This will allow us to improve our dashboards filtering out errors we are
responsible for, like missing certs on the machines.

Reviewed By: mitrandir77

Differential Revision: D19950614

fbshipit-source-id: 73503e984dfe8513a700fdcb2fc36b1618c20a4f
 2020-02-18 14:28:33 -08:00
Mark Thomas
0387f1c67e localrepo: add configurable limits for description and extras 
Summary:
Commit messages and extras can be unbounded in size.  This can cause problems if users create commits with exceptionally large messages or extras.  Mercurial will commit these to the changelog, increasing its size.  On Mononoke, large commit messages may go over the cacheing threshold, resulting in poor performance for requests involving these commits as Mononoke will need to reload on every access.

Commit messages should not usually be that large.  Mostly likely it will happen by accident, e.g. through use of `hg commit -l some-large-file`. Prevent this from happening by accident by adding configuration for soft limits when creating commits.

If a user really does need to create a commit with a very large message or extras, they can override using the config option.

Reviewed By: xavierd

Differential Revision: D19942522

fbshipit-source-id: 09b9fe1f470467237acc1b20286d2b1d2ab25613
 2020-02-18 13:12:58 -08:00
Adam Simpkins
1a263466c6 remove the unused sparsematchfn dirstate parameter 
Summary:
This parameter was originally removed in D12811551, but re-added in D12855935
due to the fact that at the time the `eden_dirstate.py` and `dirstate.py`
files were deployed in separate RPMs and could not be updated together
atomically.  We now deploy these files together, so we can drop this extra
unnecessary argument.

Reviewed By: chadaustin

Differential Revision: D19913057

fbshipit-source-id: 0f0b4fde4b3124a8fc5bb568551b4e67de14d410
 2020-02-18 11:39:22 -08:00
Jun Wu
795b679d60 doctor: support efficient treestate (dirstate) repair 
Summary:
Sometimes the treestate points to an unknown commit (ex. aborted transaction
might strip commits). While `debugrebuilddirstate -r HASH --hidden` is able to
fix it, it is too slow.

This diff adds treestate repair logic to the `doctor` command. It scans through
the treestate files, find a most recent `Root` entry with `p1` pointing to a
known commit.

This can be much faster than `debugrebuilddirstate` in some cases, because the
watchman clock might still be valid, and the NEED_CHECK file list might still
be small. In that case, `status` can still be fast.

Since treestate atomically updates all information needed for `status`
calculation (parents, need-check-files (or, "non-normal files"), watchman-clock
(only with fsmonitor), and stat for clean files). Reverting to a previous state
is still atomic. Correctness-wise, this is equivalent to aborting a "large"
transaction, and restoring treestate data to the state before the transaction.
It should be consistent, and the next `status` call won't mis-report files like
the dangerous `debugsetparents` command.

Reviewed By: DurhamG

Differential Revision: D19864422

fbshipit-source-id: d5d2f8b43a0c15ea2ac0e3c164edec7deeb8451f
 2020-02-18 09:41:32 -08:00
Jun Wu
bc7c3c1e24 doctor: remove invalid commits in visibleheads 
Summary:
See the test change. Without this change repairing the changelog won't give the
user back a working repo.

Reviewed By: markbt

Differential Revision: D19864421

fbshipit-source-id: b84582c5302469828c8cfcb3db362ea82f2eea63
 2020-02-18 09:41:31 -08:00
Jun Wu
febe96048e doctor: integrate fixcorrupt logic to repair changelog 
Summary:
Reuse utilities in the fixcorrupt extension to repair changelog.

This is better than fixcorrupt because `hg doctor` does not require a repo
object. Some messages are updated so they become more consistent with the
rest of `hg doctor`.

The main motivation is to get changelog fixed early, so other repair logic can
check if a commit hash is known by changelog or not.

Reviewed By: markbt

Differential Revision: D19864418

fbshipit-source-id: 6f95c6c6191d7db2a474a07a5278a857cf41d8e2
 2020-02-18 09:41:31 -08:00
Jun Wu
6f2b529e94 doctor: run 'edenfsctl doctor' on edenfs repos 
Summary:
Run 'edenfsctl doctor' on an edenfs repo. If there is no current repo, it might
be caused by edenfs daemon stopped running. So let's also run edenfsctl doctor
in that case.

Reviewed By: markbt

Differential Revision: D19864419

fbshipit-source-id: d2a49a126a040845b88b4883d214162326d08d8d
 2020-02-18 09:41:30 -08:00
Durham Goode
99a829340b py3: move sampling to use mercurial.json 
Summary:
We're seeing a user have issues because their username contains unicode
characters and sampling's use of json doesn't handle it well. I've not been able
to repro it unfortunately, but let's go ahead and switch sampling to use
mercurial.json.

Differential Revision: D19895419

fbshipit-source-id: a1f087d1e2c7568488c2b8d54f267bd5c8266202
 2020-02-18 09:36:28 -08:00
Xavier Deguillard
d8064b5e2a types: add a Sha256 type 
Summary: This will be used in the LFS store.

Reviewed By: DurhamG

Differential Revision: D19895803

fbshipit-source-id: 4cf447987c10fed0b5c98904f20c841428965d89
 2020-02-18 08:32:33 -08:00
Xavier Deguillard
17cc9ab5ab revisionstore: add a wrapper around IndexedLog/RotateLog 
Summary:
In some cases, higher level stores may want to store data in either a plain
IndexedLog, or in a RotateLog, for local and shared data. Due to slight
difference between the 2, they can't easily be adapted into a common trait.

Instead let's just wrap both into an enum and implement the main functions that
the higher level stores need.

The first use of this will be the LfsStore, future use will include the
IndexedLogDataStore and the IndexedLogHistoryStores.

Reviewed By: DurhamG

Differential Revision: D19859292

fbshipit-source-id: 920572e0cf5f69bda4901a727a6b0dc0f08fc8d0
 2020-02-18 08:32:32 -08:00
Durham Goode
d15cb231fb py3: get fastannotate closer to passing 
Summary: There's still some issues, but it's a lot closer.

Reviewed By: quark-zju

Differential Revision: D19802023

fbshipit-source-id: da539094cbc0ba3542e4b5fd3d49f5f80455ec23
 2020-02-17 14:52:40 -08:00
Durham Goode
09de146878 py3: fix archive 
Reviewed By: quark-zju

Differential Revision: D19802024

fbshipit-source-id: 714f1a3af7f62c8215310fe95ac4de28f9b10f1d
 2020-02-17 14:52:40 -08:00
Durham Goode
c9b1fee40f py3: fix eol 
Reviewed By: markbt

Differential Revision: D19802025

fbshipit-source-id: f6e6c03ea2d64500b92bcf8b97c9ddedb83fd3c0
 2020-02-17 14:52:40 -08:00
Durham Goode
c4f7d32501 py3: enable more tests 
Summary: Not sure what fixed these, but let's enable them.

Reviewed By: quark-zju

Differential Revision: D19800696

fbshipit-source-id: 2536a121ae37d4b23ef09dcfee8d92ccbbef3af6
 2020-02-17 14:52:39 -08:00
Durham Goode
cd8e154e36 py3: fix test-bookmarks.t 
Reviewed By: quark-zju

Differential Revision: D19800698

fbshipit-source-id: a9267e3bafa6f8abb472c0002815fdd2bf2024ca
 2020-02-17 14:52:39 -08:00
Durham Goode
779b0beb0b py3: fix test-fb-hgext-phabstatus.t 
Reviewed By: quark-zju

Differential Revision: D19800700

fbshipit-source-id: b9b052b5ad1bfa1c714b880cf79c9a21cfebadc2
 2020-02-17 14:52:39 -08:00
Durham Goode
6b84a5da81 py3: fix stablerev tests 
Reviewed By: quark-zju

Differential Revision: D19800695

fbshipit-source-id: a08efe3c43b1fa01be15ec43a6e32f68dd8eec71
 2020-02-17 14:52:38 -08:00
Durham Goode
a6353c55f5 py3: fix sparse 
Reviewed By: quark-zju

Differential Revision: D19800699

fbshipit-source-id: 7cf86d0ab9a0efc96966ac3747f56b229251fb0d
 2020-02-17 14:52:38 -08:00
Durham Goode
00d72e96bf py3: fix gitlookup 
Reviewed By: xavierd

Differential Revision: D19800701

fbshipit-source-id: ecc918a0ef7358a29789f0ab6ca50337bec9edf1
 2020-02-17 14:52:38 -08:00
Durham Goode
f05a9d79bb py3: fix test-flagprocessors.t 
Reviewed By: xavierd

Differential Revision: D19800697

fbshipit-source-id: a785a4a04ce627257993d83197fec270a8688b7d
 2020-02-17 14:52:37 -08:00
Durham Goode
fdbaaeeff9 py3: fix treedirstate unicode handling 
Summary:
There was a spot where we returned bytes for a filepath. Fix this to
make dirstate tests pass more.

Reviewed By: quark-zju

Differential Revision: D19786274

fbshipit-source-id: 7465cae8bb2e3be7758abc6279ed3f5f59581732
 2020-02-17 14:52:37 -08:00
Durham Goode
4f8c30b04e py3: enable 92 py3 tests 
Summary: These now pass

Reviewed By: singhsrb

Differential Revision: D19785175

fbshipit-source-id: bf92757e3fe0753e1b61ffddfd30a37fb40a642f
 2020-02-17 14:52:37 -08:00
Durham Goode
c2f42204cc py3: fix some mutation tests 
Reviewed By: quark-zju

Differential Revision: D19772615

fbshipit-source-id: 93bcefb6d911941a9e46107f124730279398ceff
 2020-02-17 14:52:36 -08:00
Durham Goode
5f46843003 py3: get several infinitepush tests passing 
Reviewed By: quark-zju

Differential Revision: D19772622

fbshipit-source-id: c0b64e3a22c085656dbff5526765020b64cc73f5
 2020-02-17 14:52:36 -08:00
Durham Goode
5827145ef2 py3: fix some infinitepush py3 issues 
Reviewed By: quark-zju

Differential Revision: D19772618

fbshipit-source-id: c5e520a3cec6496e24cc5b3825ef7287ab82f304
 2020-02-17 14:52:36 -08:00
Durham Goode
56f4d5ae18 py3: enable a few tests that got fixed along the way 
Summary: Not sure where these got fixed.

Reviewed By: quark-zju

Differential Revision: D19772617

fbshipit-source-id: 7bebd15ad080e1fc224d8c1e78f645877551ac92
 2020-02-17 14:52:35 -08:00
Durham Goode
2c18b7521a py3: make test-fb-hgext-remotefilelog-clone-tree.t pass 
Reviewed By: quark-zju

Differential Revision: D19772620

fbshipit-source-id: 26d1ae840e1e19c7dabbdf7a71e2d4b0d265e141
 2020-02-17 14:52:35 -08:00
Durham Goode
255b677f91 py3: fix test-fb-hgext-remotefilelog-lfs.t 
Reviewed By: quark-zju

Differential Revision: D19772621

fbshipit-source-id: 25fda0f356129b27413b25cfb70e9d08bbc56263
 2020-02-17 14:52:35 -08:00
Durham Goode
a8b4789109 py3: fix misc encoding errors that showed up in tests 
Summary:
These are a random batch of stack traces that show up in remotefilelog
tests.

Reviewed By: quark-zju

Differential Revision: D19772619

fbshipit-source-id: a8b9ce188cb7a5a2c9ccaeb62f6744f1c4083e38
 2020-02-17 14:52:34 -08:00
Durham Goode
2b0c3ec119 py3: fix test-fb-hgext-remotefilelog-histpack.py 
Reviewed By: quark-zju

Differential Revision: D19772616

fbshipit-source-id: 443527d37aa87c1802323aa6edf863beb497a327
 2020-02-17 14:52:34 -08:00
Durham Goode
a5ba21d4d4 py3: codemode assertEquals to assertEqual 
Summary:
assertEquals is deprecated and shows warnings in the tests in python 3

ignore-conflict-markers

Reviewed By: quark-zju, sfilipco

Differential Revision: D19907385

fbshipit-source-id: 7d261489856a4eeb6719eae581ed986d0415d99e
 2020-02-17 14:52:34 -08:00
Durham Goode
41cc67d011 py3: fix remotefilelog repack tests 
Reviewed By: quark-zju

Differential Revision: D19751790

fbshipit-source-id: a898a8c37929a73ee2654d2907e9ea5bb2e3fd8a
 2020-02-17 14:52:33 -08:00
Durham Goode
3a5728b32f py3: fix archive 
Summary:
archive uses a formatter to produce it's metadat file. We need to use a
string io stream instead of bytes.

Reviewed By: quark-zju

Differential Revision: D19748163

fbshipit-source-id: fbd8c32066cfc4a234d9b51691717c7fce4c7c9a
 2020-02-17 14:52:33 -08:00
Durham Goode
48aec8ca41 py3: misc fixes to remotefilelog and treemanifest 
Reviewed By: quark-zju

Differential Revision: D19747715

fbshipit-source-id: da6716ff46342d777d45bec8e560ab41d544645f
 2020-02-17 14:52:33 -08:00
Durham Goode
150aa7be3d py3: fix bundle2 bookmark default values 
Summary: They need to be bytes to match the nodes.

Reviewed By: xavierd

Differential Revision: D19746027

fbshipit-source-id: 41e9cc390f4aa97d8c8b378144e64100811665c2
 2020-02-17 14:52:32 -08:00
Durham Goode
6d40f2c380 py3: fix occasional ssh hang 
Summary:
sshpeer._calltwowaystream used iter(fp, ""), where "" was the sentinel
indicating the fp was empty. Since fp was an iterator of bytes, the sentinel was
never hit and this code kept sending 0\n to each other until the pipes filled
and it hung.

Reviewed By: xavierd

Differential Revision: D19746028

fbshipit-source-id: 0daada0ae2356b5c99cc2c39c121cd88af8f750a
 2020-02-17 14:52:32 -08:00
Durham Goode
a3ec80f7f5 py3: various fixes to move bundle2 tests foward 
Reviewed By: quark-zju

Differential Revision: D19717253

fbshipit-source-id: 3f07a3c6ff09c1f7081d2dd15659505b4a07a91a
 2020-02-17 14:52:32 -08:00
Durham Goode
67ade10253 py3: fix minor bookmark and sparse encoding issues 
Reviewed By: quark-zju

Differential Revision: D19717254

fbshipit-source-id: 1ee7fab0bec1bb5b42c2707e7bc07aa8300f4be7
 2020-02-17 14:52:31 -08:00
Durham Goode
4d4274ce53 py3: make conduithttp.py python 3 compatible 
Reviewed By: quark-zju

Differential Revision: D19717252

fbshipit-source-id: 075aaed642898a3649d40f9cdc9752539b0970aa
 2020-02-17 14:52:31 -08:00
Durham Goode
66fd50346c py3: add pycompat buffer alias 
Summary: buffer in Python 2 became memoryview in Python 3.

Reviewed By: quark-zju

Differential Revision: D19717251

fbshipit-source-id: f678abfed7e2ec94053329b13015bff3c31eb319
 2020-02-17 14:52:30 -08:00
Durham Goode
e4b5247311 py3: fix test-cat.t 
Summary:
Fixes test-cat.t by changing some formatter encoding choices. The
formatter is bit awkward here.  hg cat needs to support outputting raw binary,
but also json and templated output.

For now I've set it up so json and templated output can't output non-utf8 data.

Reviewed By: quark-zju

Differential Revision: D19786542

fbshipit-source-id: 84060928103b396b23e3173b715aed996074fa3e
 2020-02-17 14:52:30 -08:00
Durham Goode
734ce7959f py3: fix various lfs encoding issues 
Reviewed By: quark-zju

Differential Revision: D19717255

fbshipit-source-id: d435a65e566b3a74399632e52c7cd99d0bba7f0a
 2020-02-17 14:52:30 -08:00
Durham Goode
9d09b429f0 py3: migrate treedirstate to use utf8 for state and files 
Summary:
treedirstate was using bytes for state and for file paths. This does
the appropriate conversions. Note, I don't use strings for all state in rust,
because it's a pain in the butt in some places. We're going to delete
treedirstate eventually anyone, so just I'm getting the test to pass for now.

Reviewed By: quark-zju

Differential Revision: D19715102

fbshipit-source-id: 4f9eac372bee7884d36aa19e3a3ed253392fa7dd
 2020-02-17 14:52:29 -08:00
Durham Goode
f530333e06 edenfs: update eden thrift types 
Summary:
When I run make local it's creating changes in our checked in thrift
types. I guess I need to check these in?

Reviewed By: quark-zju

Differential Revision: D19848706

fbshipit-source-id: 8a2e9a2617734eda41eade1f2645689362b1d75d
 2020-02-17 14:52:29 -08:00
Mark Thomas
892ac0079e rage: add debugmutation for recent draft commits 
Summary:
Add `debugmutation` output for recent draft commits to `hg rage`.  This will
allow us to easily see the recent history of the draft commits in a user's
repo.

Reviewed By: DurhamG

Differential Revision: D19903999

fbshipit-source-id: f6ebd729812c63d3760f9dab031414df6b24ab28
 2020-02-17 06:41:04 -08:00
Mark Thomas
3690e8ab44 debugmutation: add time range limiting 
Summary:
Make it possible to limit the time range of mutation info being displayed by hg
debugmutation.

Reviewed By: DurhamG

Differential Revision: D19904000

fbshipit-source-id: 365f54fdd861661961bba1a0ea96fce772623a23
 2020-02-17 06:41:03 -08:00
Mark Thomas
5122958c70 commitcloud: fix program name template in commit cloud warning 
Summary: The correct template for the program name is `prog@`.

Reviewed By: HarveyHunt

Differential Revision: D19904938

fbshipit-source-id: 19a3ac18f85e2b090f53d0423fce3c6982cf32fd
 2020-02-17 02:37:26 -08:00
Xavier Deguillard
7bb3e384d8 remotefilelog: append the repo name to memcache key 
Summary:
Up to now, this has been done in chef, and thus for repos that we do not list,
they may share the memcache keys, with potential unintended consequences. Let's
always add the repo name to the key, so we can simplify the code in chef.

One small negative effect of this change is that while it is being rolled out,
the cache hit rate will be impacted. This should resolve itself quickly.

Reviewed By: DurhamG

Differential Revision: D19885775

fbshipit-source-id: 0b59ce9e378b0ab70f696a39d19d27cd89921098
 2020-02-14 14:10:48 -08:00
Mark Thomas
7f71cd1cda commands: pass date-based rev as a hex hash 
Summary:
When using the `--date` parameter to `hg update` or `hg revert`,
`cmdutil.finddate` returns the binary node for the target commit.  Passing this
to `scmutil.revsingle` sometimes works, however it's unreliable, as if the
binary node happens to look vaguely like a revset, we will attempt to parse it,
and perhaps fail.

Resolve the ambiguity by converting the binary node to hex.

Reviewed By: quark-zju

Differential Revision: D19902595

fbshipit-source-id: 5eb7b9b029e292a02ccc00f5a465ab7807cd56d5
 2020-02-14 09:39:55 -08:00
Xavier Deguillard
28564b228d backingstore: do not fail if memcache can't be initialized 
Summary:
Failing means that we fallback to the Python importer. Let's simply warn about
it.

Reviewed By: fanzeyi

Differential Revision: D19897274

fbshipit-source-id: f9c63f5aa76015c28b31f00bba98244f5c86e923
 2020-02-14 09:00:27 -08:00
Jun Wu
a8874af094 run-tests: clear RUST_BACKTRACE and RUST_LIB_BACKTRACE 
Summary:
Some Mononoke tests test about the backtrace. Having the environment variables
set break those tests. Clear them.

Reviewed By: xavierd

Differential Revision: D19887219

fbshipit-source-id: 529a17282c40730ee95bffbed00070edd1f0823a
 2020-02-13 14:50:51 -08:00
Jun Wu
116f5d5451 distutils_rust: retry on mt.exe exitcode 31 error 
Summary:
`mt.exe` can fail when Windows Anti-Virus scans the same file. Retry on such
failures. This hopefully can improve our build success rate on the Windows
platform.

Reviewed By: xavierd

Differential Revision: D19887220

fbshipit-source-id: b725d5fe883bd52697b58b74dbb4a338c02c3ed6
 2020-02-13 14:50:51 -08:00
Jun Wu
4ec1633c15 test-fb-hgext-treemanifest-server: attempt to fix '[[' not found error 
Summary:
This is an attempt to fix the following error:

```
 --- test-fb-hgext-treemanifest-server.t
+++ test-fb-hgext-treemanifest-server.t.simplecachestore.err
@@ -167,8 +167,9 @@
   $ hg push --to mybook
   pushing to ssh://user@dummy/master
   searching for changes
-  remote: prepushrebase.myhook hook exited with status 1
+  remote: prepushrebase.myhook hook exited with status 2
   abort: push failed on remote
+  remote: $TESTTMP/myhook.sh: 2: $TESTTMP/myhook.sh: [[: not found
   [255]
```

which I suspect is caused by a non-bash default shell.

Reviewed By: xavierd

Differential Revision: D19887222

fbshipit-source-id: b6fe5d89e4c41ff49fca86da927c4e702ed1e7c1
 2020-02-13 14:50:51 -08:00
Jun Wu
d1750883cb py3: sshserver: detect client close correctly 
Summary:
This is needed by the next diff. Otherwise Python 3 ssh tests in the next diff
will hang.

Reviewed By: DurhamG

Differential Revision: D19879882

fbshipit-source-id: ecc317d0685993c6b1bef8c72068bf4315030d0f
 2020-02-13 12:16:22 -08:00
Jun Wu
594f668295 connectionpool: add a debug note saying connection being reused 
Summary:
I'm going to change the connection pool logic but I'm not sure where it gets
used. This change exposes at least one test using it.

Reviewed By: xavierd

Differential Revision: D19872614

fbshipit-source-id: 4921b92c3fe3fd7ba1a72de17eef92604964eb2e
 2020-02-13 12:16:21 -08:00
Jun Wu
004ebb0ddc tests: fix test-fb-hgext-lfspushrebase-verify-blobs 
Summary:
Regressed since D19702533.  This test breakage is only visible if
lfs-test-server exists.

Reviewed By: farnz

Differential Revision: D19872608

fbshipit-source-id: 74ace3eb7363bb1bb773e6b448685e9a3874086f
 2020-02-13 07:51:26 -08:00
Jun Wu
60e4f31be9 test-ssh: run contrib/hg-ssh with hg debugpython 
Summary: The script does not run with the stock python since it depends on edenscm.

Reviewed By: farnz

Differential Revision: D19872953

fbshipit-source-id: 4c5b2d2935a0c9e8cf0a654f541600d7a4fd7211
 2020-02-13 07:51:25 -08:00
Kostia Balytskyi
a3f1d6ae64 clienttelemetry: report multiple server_hostnames 
Summary:
Right now, if the client establishes connection to more than one peer, the
last one to proces `clienttelemetry` wireproto command gets the honor to set
`server_realhostname`. This is not desirable. Specifically, when we have
`fallbackpath` set up for remotefilelog/treemanifest and the prefetch happens
after pulling a commit, we get the hostname of the fallbackpath server,
while losing the hostname of the original `getbundle` server.

Reviewed By: DurhamG

Differential Revision: D19837570

fbshipit-source-id: fdc41565a5dfe670df3caf3b034196c4b7bdf6d9
 2020-02-13 00:48:25 -08:00
Stanislau Hlebik
12c2453526 remotenames: log when a node is missing 
Reviewed By: DurhamG

Differential Revision: D19832734

fbshipit-source-id: 2861b05cc754b3e3865dee706290fe3bed08d336
 2020-02-13 00:23:54 -08:00
Wez Furlong
90dbc47c5a eden: apfs: add delete-all subcommand 
Summary: It is helpful in CI to be able to remove all eden managed volumes at the end of a job, so that is what this command does.

Reviewed By: simpkins

Differential Revision: D19794482

fbshipit-source-id: d832d093d0a6369a4b533d66afbdce626fa78e2c
 2020-02-12 17:59:05 -08:00
Wez Furlong
9423f8d70a eden: apfs: fixup mount point detection 
Summary:
In the initial iteration of eden_apfs_mount_helper I parsed the human
readable version of the `diskutil apfs list` output to get all of the useful
information from it.  Later, I switched to using `diskutil apfs list -plist`
because it simplified the parsing dramatically.

However, it was overlooked that the `mount_point` field was not present in
the plist output.  That was usually fine, except in the one case that I
didn't test after changing the parser: after reboot macOS picks a volume
name to remount these volumes.  The lack of the mount_point field meant
that we would simply skip the logic that deals with unmounting and remounting
the volume in the correct place.

In order to address this gap we now need to parse the mount table to determine
where the device is mounted, so that is what this diff does.

Reviewed By: simpkins

Differential Revision: D19794478

fbshipit-source-id: ace2df145a46aad7df78c3f4b15fb2198aef3e6f
 2020-02-12 17:59:05 -08:00
Wez Furlong
90a69e77ca eden: apfs helper: disable spotlight, fsevents, trashes 
Summary:
This is done on a sort of best effort basis; the problem
we're aiming to solve (or at least avoid in the majority of cases)
is in these scenarios:

* `buck clean` gives up cleaning as soon as it finds a directory that
  it doesn't have permissions to remove.
* `yarn` tries to look for node modules inside the `.Trashes` dir,
  which it doesn't have permissions to access

macOS doesn't give us a solid way to indicate that these things should
be disabled at the time that we mount the volume; the various docs
and suggestions online all involve creating marker files to block the
system processes from performing their usual actions, and we could
set those up here, but if we create them with root permissions we'll
trigger the bad behavior in buck and yarn.  If we create them with
regular user permissions then eg: running buck clean would remove them
and allow the system to recreate them.

The system will recreate the `.Trashes` directory when the user sends
something to the trash via Finder.   Similarly, macOS will recreate
the fsevents directories when an application establishes a watch
on the volume using fsevents.

So we can't permanently prevent this problem, but by deleting these
things at mount time we should at least avoid it bubbling up for
the majority of users, and if those things come back and get in the
way, running `eden redirect unmount ; eden redirect fixup` should
remount and re-remove the problematic things.

Reviewed By: fanzeyi

Differential Revision: D19841514

fbshipit-source-id: f530ab3d68edfa643096bd27efae71c80b505184
 2020-02-12 17:59:04 -08:00
Jun Wu
03baa31789 indexedlog: switch from bytes to minibytes 
Summary:
This makes it possible to use `Bytes` for mmap buffers.

The changes are because `minibytes::Bytes` does not implement `From<&[u8]>`
with the intention to make slice copy explicit.

Reviewed By: xavierd

Differential Revision: D19818719

fbshipit-source-id: c34ee451bfd2dc7bcbbcebd52a76444b6c236849
 2020-02-12 13:57:37 -08:00
Xavier Deguillard
49c953bc7e backingstore: plumb the MemcacheStore 
Summary:
EdenFS will now be able to fetch blobs directly from memcache. This won't have
any big benefits as no blobs are in memcache right now, but over time, this
will significantly reduce the cost of fetching blobs.

Reviewed By: fanzeyi

Differential Revision: D19861643

fbshipit-source-id: c2e9d317bd30d4656bf0b3f8897794161697761a
 2020-02-12 13:43:00 -08:00
Xavier Deguillard
a4b83e384a revisionstore: add tracing point for memcache 
Summary:
These tracing points will help us understand the memcache hit rate as well as
the fetching speed.

Reviewed By: quark-zju

Differential Revision: D19836499

fbshipit-source-id: 1936c44efc3e7715069e6a959f5331139d591d5c
 2020-02-12 10:38:59 -08:00
Xavier Deguillard
2c4a10bf4b revisionstore: move memcache set to a background thread 
Summary:
Everytime a cache miss is seen, the data fetched from the server will be sent
directly to memcache for future use. Unfortunately, doing so in a blocking
manner severely impact the overall fetching speed from the server. Since
memcache is purely an optimization, we can afford to send data to it
asynchronously.

Let's move as much as possible of the code to a background thread to reduce the
overhead of memcache.

Reviewed By: DurhamG

Differential Revision: D19836011

fbshipit-source-id: 68e506ef7464d6e99d98457d0d37178f514be1a9
 2020-02-12 10:38:59 -08:00