Commit Graph

193 Commits

Author SHA1 Message Date
Durham Goode
c2f42204cc py3: fix some mutation tests
Reviewed By: quark-zju

Differential Revision: D19772615

fbshipit-source-id: 93bcefb6d911941a9e46107f124730279398ceff
2020-02-17 14:52:36 -08:00
Durham Goode
41cc67d011 py3: fix remotefilelog repack tests
Reviewed By: quark-zju

Differential Revision: D19751790

fbshipit-source-id: a898a8c37929a73ee2654d2907e9ea5bb2e3fd8a
2020-02-17 14:52:33 -08:00
Durham Goode
9d09b429f0 py3: migrate treedirstate to use utf8 for state and files
Summary:
treedirstate was using bytes for state and for file paths. This does
the appropriate conversions. Note, I don't use strings for all state in rust,
because it's a pain in the butt in some places. We're going to delete
treedirstate eventually anyone, so just I'm getting the test to pass for now.

Reviewed By: quark-zju

Differential Revision: D19715102

fbshipit-source-id: 4f9eac372bee7884d36aa19e3a3ed253392fa7dd
2020-02-17 14:52:29 -08:00
Xavier Deguillard
b8947748b5 pyrevisionstore: expose the memcache client to python
Summary:
This allows the Python code to build a memcache client and build ContentStore
and MetadataStore with it.

Reviewed By: DurhamG

Differential Revision: D19518694

fbshipit-source-id: d932fd5223ccfdf37db69cbb54a11a6571312709
2020-02-05 14:01:54 -08:00
David Tolnay
d358c71e6d Update to Rust 1.41.0
Reviewed By: jsgf

Differential Revision: D19656956

fbshipit-source-id: ef791638021ae55375ca35ca191cf1b8a3cc5d7f
2020-02-04 18:03:03 -08:00
David Tolnay
34a520536a Update rustfmt and reformat fbsource
Summary:
```
$ tools/third-party/rustfmt/rustfmt --version
rustfmt 1.4.11-nightly (1838235 2019-12-03)
```

Reviewed By: zertosh

Differential Revision: D19704678

fbshipit-source-id: fe8707e964495e76746edcb8b68e34fc1411f52a
2020-02-04 17:14:27 -08:00
Xavier Deguillard
fe3fc9712f pyrevisionstore: copyfrom should be PyPathBuf, not PyBytes
Summary: This was missed while converting pyrevisionstore to PyPathBuf.

Reviewed By: markbt

Differential Revision: D19672441

fbshipit-source-id: 466a66f2e00c7f73c11a8989c22508560f423e0e
2020-01-31 16:19:25 -08:00
Mark Thomas
8e673a1de0 py3: implement util.dirs in Rust
Summary:
The C implementation of util.dirs doesn't work with Python 3.  Instead of trying to port it, write an implementation in Rust that works in both Python 2 and Python 3.

The old implementation had a hack for flat dirstate to omit certain entries.  Remove this hack as we now only use flat dirstate in tests and small repos.  This simplifies the code in the fastpath case.

Reviewed By: quark-zju

Differential Revision: D19634699

fbshipit-source-id: cbb6562bc9a0209b13f67db3dfa5250800a9fcbe
2020-01-30 17:33:36 -08:00
Mark Thomas
914607cac7 cpython-ext: add PyPath for references to paths
Summary: `PyPath` is to `PyPathBuf` as `Path` is to `PathBuf` and `str` is to `String`.

Reviewed By: quark-zju

Differential Revision: D19647995

fbshipit-source-id: 841a5f6fea295bc72b00da028ae256ca38578504
2020-01-30 17:33:35 -08:00
Arun Kulshreshtha
dd34889d43 py3: make patchrmdir work with Python 3
Summary: With this change `hg rm` works even with extensions enabled. Since this is a Cython extension, I did not bother adding type annotations since IIUC Pyre cannot type check Cython files.

Reviewed By: quark-zju

Differential Revision: D19652178

fbshipit-source-id: 2e03a0f241f80732bdae3f472a1e0576684b1911
2020-01-30 17:22:30 -08:00
Durham Goode
b567c16b60 py3: make mutation markers 'user' utf8
Summary: Username as utf8, so let's make mutationmarker treat them as such.

Reviewed By: xavierd

Differential Revision: D19649887

fbshipit-source-id: 3f8b2db434a57ee8ee3017de8d925c19a2002b20
2020-01-30 15:22:24 -08:00
Xavier Deguillard
b863a561a4 pytreestate: use PyPath where necessary
Summary:
A previous diff missed these PyPath. As a bonus, replace most of the PyObject
into proper Rust types.

Reviewed By: quark-zju

Differential Revision: D19646139

fbshipit-source-id: a3b55e5a802d8bbef86e9a9431f605bb4e284b32
2020-01-30 14:55:54 -08:00
Stefan Filip
843f8a7f50 py3/cython: set the language_level to 3str
Summary:
Follow up to the previous diff setting language_level to 3str for
`edenscmnative/clindex.pyx`. Consistent behavior for all files.

Reviewed By: quark-zju

Differential Revision: D19634907

fbshipit-source-id: 779d6090b5660e109a37f9740299fb9623d8f34d
2020-01-30 12:52:06 -08:00
Mark Thomas
13b7a759a2 cpython-ext: add PyNone, a marker struct for functions that can only return None
Summary: Add `PyNone`.  This is a marker struct that indicates that a python function can only return `PyNone`.

Reviewed By: xavierd

Differential Revision: D19644338

fbshipit-source-id: f846b146237ebf7de996177494934fec662cde0f
2020-01-30 12:28:38 -08:00
Mark Thomas
6b8042662a cpython_ext: rename PyPath to PyPathBuf
Summary:
`PyPath` is the type that owns the data.  Rename it to `PyPathBuf` for analogy
with `PathBuf` and `RepoPathBuf`, and to allow us to introduce a reference type
named `PyPath`.

Reviewed By: xavierd

Differential Revision: D19643797

fbshipit-source-id: 56d80fea5677f7223e967b0723039d1763b26f68
2020-01-30 11:06:24 -08:00
Xavier Deguillard
f83e1d76b0 pypathmatcher: use PyPath when calling into Python
Summary: The Python code expect paths to be unicode in Python3, not bytes.

Reviewed By: markbt

Differential Revision: D19643701

fbshipit-source-id: fa98da2daccccf70ac047a69b2e2359270f7e3d0
2020-01-30 10:55:34 -08:00
Xavier Deguillard
a14897afc8 clindex: remove one b
Summary: In `hg sl`, this raises an exception.

Reviewed By: singhsrb

Differential Revision: D19635386

fbshipit-source-id: 28482941bd61b5526993d6d16b2886c7edbe763f
2020-01-30 08:31:23 -08:00
Jun Wu
097503d073 py3: treestate: use IntoAnyhowResult
Summary: This preserves the Python errors across language boundaries.

Reviewed By: xavierd

Differential Revision: D19635355

fbshipit-source-id: 14ab8af38362182ca43951d4c560fa405687bf6e
2020-01-30 08:27:34 -08:00
Jun Wu
002d32e878 py3: treestate: use PyPath
Summary: Use PyPath for input/output and Python callbacks.

Reviewed By: xavierd

Differential Revision: D19635356

fbshipit-source-id: c2d8898ef903fda436277997a842bc837bb1bf2f
2020-01-30 08:27:33 -08:00
Adam Simpkins
b9790d38d6 py3: explicitly set the language_level to 3str
Summary:
This causes the code to always treat unprefixed string literals as the `str`:
i.e., bytes in Python 2 and Unicode in Python 3.  Without this setting our
unprefixed string literals were always being treated as unicode.

Reviewed By: lisroach

Differential Revision: D19634266

fbshipit-source-id: 770d7d03a96aa92a4e183819d34c72281c6c794b
2020-01-29 17:46:30 -08:00
Stefan Filip
49e90ce242 py3/pypathmatcher: return Str instead of Bytes
Summary:
Globs and patterns generally speaking. They should be typed as Str in
Python3 following how paths are typed.

Reviewed By: quark-zju

Differential Revision: D19628654

fbshipit-source-id: eeaede72f41258d6fe0f3cde26c434fe0b908b8b
2020-01-29 17:03:04 -08:00
Xavier Deguillard
73d424dd90 pyedenapi: use PyPath and String instead of PyBytes
Summary: This allows edenapi to be able to fetch trees on Python3.

Reviewed By: quark-zju

Differential Revision: D19629411

fbshipit-source-id: f6abd585d8d672854f7219ef8e3ab86d2f659118
2020-01-29 14:51:43 -08:00
Jun Wu
c18818de39 py3: pyindexes: use str for hex prefix matching
Summary: The hex prefix should be in str, not bytes.

Reviewed By: mitrandir77

Differential Revision: D19627762

fbshipit-source-id: dc9b054198dc61614518fec9bba69f789326ff8b
2020-01-29 13:43:09 -08:00
Xavier Deguillard
f2733fb294 pyrevisionstore: use PyPath for key name
Summary:
The names are really RepoPath, and thus let's use PyPath as a way to pass these
in.

Reviewed By: quark-zju, sfilipco

Differential Revision: D19625372

fbshipit-source-id: 4802030b91e6a065d3cb1905c770cad8a86da510
2020-01-29 13:32:46 -08:00
Xavier Deguillard
20459835bb clindex: use str as input to vfs
Reviewed By: farnz

Differential Revision: D19624129

fbshipit-source-id: cbee989334d5cf114ef7b74c9143a992cee70370
2020-01-29 11:07:09 -08:00
Xavier Deguillard
776f0fabac pyedenapi: use String for the client constructor
Summary:
Both the repo name and the url come from the config, and thus Unicode values
are passed in in Python3. The String type handles the Python2/Python3
difference, so let's use it instead of PyBytes.

Reviewed By: DurhamG

Differential Revision: D19623202

fbshipit-source-id: 72c72f8338fbb6a2f3e88395f3587a5b59389711
2020-01-29 11:07:09 -08:00
Mark Thomas
1e63f205f4 rust-cpython: allow compilation for both py2 and py3
Summary:
Set up the `cpython-ext` and `hgcommands` libraries so that they can compile
against py2 and py3 versions of rust-cpython.  Make py2 the default so
that cargo test still works.

Reviewed By: singhsrb

Differential Revision: D19615656

fbshipit-source-id: 3403e7077deb3c0a9dfe0e3b7d4f4ad1da73bba3
2020-01-28 20:17:20 -08:00
Xavier Deguillard
40d31b3060 traceprof: remove some b
Summary:
The config functions takes Str (ie: bytes in Python2, Unicode in Python3), the
b is uncessary.

Reviewed By: sfilipco

Differential Revision: D19611274

fbshipit-source-id: 0807bd6aed1b5bd0baa0b78d3dfcbb9c46aeff35
2020-01-28 17:07:36 -08:00
Xavier Deguillard
b1e73324ea clindex: remove some b
Summary: All of these should just plain strings.

Reviewed By: sfilipco

Differential Revision: D19611275

fbshipit-source-id: 1d16c10df6f67cee9dadd7480c40950c0abae926
2020-01-28 17:07:36 -08:00
Mark Thomas
82715fd2ea py3: iter{keys,values,items} -> pycompat.iter{keys,values,items}
Reviewed By: quark-zju

Differential Revision: D19608323

fbshipit-source-id: dd186ef16d6422a56af41fcaa850d9838ae9a240
2020-01-28 16:27:28 -08:00
Xavier Deguillard
08f8dbab46 pycliparser: use Str instead of Bytes
Reviewed By: quark-zju

Differential Revision: D19607631

fbshipit-source-id: 472b660c9dcd2d044567ce091bf7fa0fdf157121
2020-01-28 15:02:21 -08:00
Adam Simpkins
22c3c06c9e py3: some additional string encoding and type issues
Summary:
Deal with a few more minor string/bytes issues in Python 3, and also drop a
reference to the `long` type, which no longer exists in Python 3.

Reviewed By: xavierd

Differential Revision: D19596744

fbshipit-source-id: d7dca9306934aa6bf8c17bada6f6c1a40130027f
2020-01-28 14:32:31 -08:00
Jun Wu
8703970cea py3: update Cargo.toml to make py3 buildable
Summary:
This makes `make hg3` work. It requires cleaning up the `build` directory when
switching between py2 and py3 build, which will be fixed later.

Reviewed By: DurhamG

Differential Revision: D19604824

fbshipit-source-id: 060ff313420126a5dba935c4451b45dc9af45f13
2020-01-28 13:39:38 -08:00
Xavier Deguillard
fcafda075f pyworkingcopy: use PyPath instead of PyByte
Reviewed By: DurhamG

Differential Revision: D19601926

fbshipit-source-id: 847e1762062c2b12627dd07a50a6d77ada282bdd
2020-01-28 12:40:49 -08:00
Xavier Deguillard
d087e39a34 pypathmatcher: use PyPath instead of PyByte
Reviewed By: DurhamG

Differential Revision: D19592136

fbshipit-source-id: 5db6ca629cd920d52ffbf7f10963c44c8f7b203d
2020-01-28 12:40:48 -08:00
Adam Simpkins
f0e087e898 py3: update pyconfigparser bindings to return results as Str
Summary:
Update the pyconfigparser library so that it returns config values as
`unicode` in Python 3.

Most of the Python code already accesses config items with `str` keys,
but a few `.pyx` files were explicitly using bytes and needed to be updated to
use string keys.

Reviewed By: xavierd

Differential Revision: D19596746

fbshipit-source-id: fbeb363639b2256c57311bea996c4b371e068ae8
2020-01-28 12:25:39 -08:00
Adam Simpkins
563ff09e85 py3: update configparser to return errors as Str
Summary:
Update the configparser APIs to return errors as a list of `Str` rather than
`Bytes`.  The Python code already expects the errors as `str`, so this makes
the code do the right thing with Python 3.

Reviewed By: xavierd

Differential Revision: D19596745

fbshipit-source-id: 027c33d76b3a3b800de34366232285591841a900
2020-01-28 12:25:39 -08:00
Mark Thomas
4fe02f3607 bindings: update to rust-cpython 0.4
Summary: Update to the new version of rust-cpython.  This supports `list.append`, so make use of it.

Reviewed By: xavierd

Differential Revision: D19590905

fbshipit-source-id: 03609d4f698ae8e4380e82b8144caaa205b4c2d4
2020-01-28 10:46:33 -08:00
Stefan Filip
5720b9a2a1 py3/pymanifest: convert path types from PyBytes to PyPath
Reviewed By: xavierd

Differential Revision: D19594134

fbshipit-source-id: e8532a125aa2ed4b7740e669ad572fcbb327692f
2020-01-28 10:29:11 -08:00
Xavier Deguillard
283b120bb6 pyconfigparser: use PyPath instead of PyByte
Summary:
Also, add a util::path::strip_unc function that is more clear than the
normalize_for_display

Reviewed By: DurhamG

Differential Revision: D19595961

fbshipit-source-id: 330bcb708bf64320a3562d79db685d6cb1e14f16
2020-01-28 10:14:14 -08:00
Xavier Deguillard
f3b73fe8bf pymetalog: remove unecessary encoding deps
Summary: The crate doesn't depend on it, let's remove the dependency

Reviewed By: quark-zju

Differential Revision: D19593138

fbshipit-source-id: ad1e9fc68f8d126b6402efd5b1be4633fc7e5887
2020-01-28 10:01:52 -08:00
Xavier Deguillard
b45d97f7df pybookmarkstore: use PyPath instead of PyByte
Reviewed By: DurhamG

Differential Revision: D19593033

fbshipit-source-id: 5e2d5b9e72c336e6290c8359071d086a15bd05f6
2020-01-28 10:01:52 -08:00
Xavier Deguillard
1ae9196972 pyblackbox: use PyPath instead of PyByte
Reviewed By: DurhamG

Differential Revision: D19592901

fbshipit-source-id: ff875ec30bee5ac160ac701091835adbf3013b90
2020-01-28 10:01:52 -08:00
Xavier Deguillard
2d2d63544d pymutationstore: use PyPath instead of PyByte
Reviewed By: DurhamG

Differential Revision: D19592467

fbshipit-source-id: 2d7df3457a9dd02e839cffb642401575712f0f2c
2020-01-28 10:01:51 -08:00
Xavier Deguillard
962fce975d pytreestate: use PyPath instead of PyBytes
Reviewed By: DurhamG

Differential Revision: D19591844

fbshipit-source-id: be2fcdc872f57307e20e24f714b05a6f461e0594
2020-01-28 10:01:51 -08:00
Xavier Deguillard
3e27c70d42 pynodemap: use PyPath instead of PyBytes
Reviewed By: DurhamG

Differential Revision: D19591313

fbshipit-source-id: bd3a8b04b83c226f23ad6d0e1b9509bf65db2f69
2020-01-28 10:01:51 -08:00
Xavier Deguillard
8d20878c1a pydag: use PyPath instead of PyBytes
Reviewed By: quark-zju

Differential Revision: D19591260

fbshipit-source-id: 72ea5be6f79d02cce22c7a5f389cf13a9c69f969
2020-01-28 10:01:50 -08:00
Xavier Deguillard
a0d934ecc2 pyedenapi: use PyPath instead of PyBytes
Reviewed By: quark-zju

Differential Revision: D19591173

fbshipit-source-id: 1a5362c99c37f342b1aeb2d95573e4d09c8440c5
2020-01-28 10:01:50 -08:00
Xavier Deguillard
61aaf894c3 pyrevisionstore: use PyPath instead of PyBytes
Summary:
For Python3 compatibility, let's use PyPath, it hides the logic of encoding for
Python2

Reviewed By: DurhamG

Differential Revision: D19590024

fbshipit-source-id: 7bed134a500b266837f3cab9b10604e1f34cc4a0
2020-01-28 10:01:50 -08:00
Stefan Filip
d78982a6e8 dag: move iddag to own file
Summary:
Separate Segment and IdDag in two individual files. This is preparation for
refactoring IdDag to be more flexible in terms of storage. That will probably
involve moving stuff out of IdDag into a new file that deals with the storage
abstractions.

Reviewed By: quark-zju

Differential Revision: D19559127

fbshipit-source-id: b3b9b18e2653157e69148b1f29292a57b30016ec
2020-01-24 15:49:54 -08:00
Xavier Deguillard
b6589bde84 revisionstore: prefetch takes &[Key] instead of Vec<Key>
Summary: This can prevent potential moves and clones on the caller of prefetch.

Reviewed By: quark-zju

Differential Revision: D19518697

fbshipit-source-id: 63839fc3f4bb9ca420e290eabaffb481a3584f7b
2020-01-23 08:57:22 -08:00
Xavier Deguillard
d88b2fed9b remotefilelog: properly support force prefetch
Summary:
There are few cases where we need to prefetch data that is already present in
the local store, but the server holds more up-to-date data. For instance, after
pushing a change to a pushrebase server, the linkrev for the just pushed commit
is no longer valid, and the history data is re-fetched from the server to fix
it.

For now, let's keep the fix for this contained in Python as we will move away
from linkrev altogether in the future, by then, this change will be unecessary
and can be reverted.

I'm overall not a big fan of the hacks needed here, suggestions are welcomed for
a cleaner way to achieve this :)

Reviewed By: DurhamG

Differential Revision: D19412620

fbshipit-source-id: 331f3e4dbb51bcd687149370a62c60c325534f60
2020-01-16 09:41:41 -08:00
Xavier Deguillard
1e809fc681 revisionstore: allow building a {Content,Metadata}Store without local stores
Summary:
In order to avoid pathological linkrevfixup after pushing a change to a
pushrebase repo, we need to be able to fetch history data that is already
present locally from the server. Since the Rust stores will always check
whether the data is present locally before fetching it, we would not be
fetching anything, causing the pathological linkrevfixup to kick in.

By allowing the stores to be built without a local component, the prefetch
function will not find the local history, and will thus be able to fetch it
properly.

Reviewed By: DurhamG

Differential Revision: D19412619

fbshipit-source-id: 421c59c63634ead7f98e6ba89da0532067f7b412
2020-01-16 09:41:40 -08:00
Xavier Deguillard
d89eab8078 pyconfigparser: use String as arguments instead of PyBytes
Summary:
In the cpython bindings, the Rust String can take both PyBytes and PyUnicode
strings, which is perfect for Python3 compatibility as string literals are
PyBytes in Python2 and PyUnicode in Python3. The return values are kept as
Bytes for now as changing this is a much larger change in itself.

Other approaches tried:
 - Using PyUnicode as input/output: an extremely large codemod had to be done,
   with very little benefits
 - Using String as output: since we do have some configs that are unicode, on
   the Python side, the output might either be bytestrings or unicode, leading
   to weird bugs.

Reviewed By: DurhamG

Differential Revision: D18650466

fbshipit-source-id: aebdf30590dcae40b7df2787e5ece88e2ec9395c
2020-01-16 09:31:45 -08:00
Mark Thomas
eef0c650e7 mutation: make the mutationstore the source of truth for mutation entries
Summary:
Previously the mutation commit extras were the source of truth for mutation
information, and the mutation store served as a kind of cache.  This turned out
to be less useful than expected, as oftentimes commits are missing, and the
store is better indexed, so in practice using the store as the source of truth
is better.

This change makes the mutationstore the (sole) source of truth for mutation
data.  The extras are kept, but they are now only useful as human-readable
debug information, and to ensure the commit hash is unique.

Collecting the mutation information during commit creation is now done through
a new `mutinfo` object.  This is a dict with the same keys as the mutation
extras, for simplicity, but it is now passed through the `committablectx` and
used to generate the mutation store entry directly.

The `mutation.enabled` config option is now used to control all aspects of
enabling mutation.

The `mutation.record` config option is now only used to indicate whether the
mutation extras should also added to the commit.  Generally this should be set
to `true`, however the option is retained so that mutation extras can be
stripped by running `hg amend --config mutation.record=false`, which no longer
has the side-effect of not recording mutation information to the store.

The "remote commit" mutation record origin is now obsolete, and won't be
generated anymore.

Pushrebase now relies on the obsmarker information coming back from the server
in order to correctly generate mutation information.  We will need to change
this so that the server returns mutation records before we can fully deprecate
obsmarkers.

Reviewed By: DurhamG

Differential Revision: D19410650

fbshipit-source-id: 8d7094e4bfd8d8e97916898d899a8debd339485f
2020-01-16 01:03:22 -08:00
Xavier Deguillard
3030ad82b0 remotefilelog: fix remotefilelog.cachepath difference for Rust ContentStore
Summary:
The error message and the exception type are slightly different between the
Rust and Python ContentStore. For now, let's just fix this up manually.

Reviewed By: quark-zju

Differential Revision: D19394350

fbshipit-source-id: e432094a9dfcf605568a1890c0303b733e98d203
2020-01-15 15:55:10 -08:00
Jun Wu
3c5766d59e dag: rename Dag to IdDag
Summary: This is more consistent with the name "NamedDag".

Reviewed By: singhsrb

Differential Revision: D19405472

fbshipit-source-id: f7023307acaf96bf77c9fa9704dcaf6fc59b56f2
2020-01-15 14:02:06 -08:00
Durham Goode
1c573e7569 workingcopy: add error handling and reporting to rust walker
Summary:
Previously, if the rust walker encountered any issues with individual
files it would stop and throw an exception or in some cases panic. To replace
the python walker we need to handle errors more gracefully, so let's add logic
for reporting which files had issues and what those issues were.

This also contains a minor one-line fix to prevent the walker from traversing
directories that contain a '.hg' directory.

Reviewed By: quark-zju

Differential Revision: D19376540

fbshipit-source-id: ede40f2b31aa1e856c564dd9956b9f593f21cf27
2020-01-13 15:26:30 -08:00
Mark Thomas
f395961e33 renderdag: add additional glyph sets for box-drawing
Summary:
Box-drawing characters with curves aren't reliably renderable on Windows.  Add
a "square" glyph set that uses right-angle characters.  These characters are
available in cp437 and cp850, so should be available on most Windows systems.

For completeness, add a "dec" glyph set that uses DEC line drawing characters,
for use in old terminals like xterm.

Reviewed By: quark-zju

Differential Revision: D19371722

fbshipit-source-id: 35887243cceab66c702e2b5278b572f77946805f
2020-01-13 10:22:42 -08:00
Jun Wu
674b592596 clindex: fix error type in partialmatch
Summary:
The expected error type is RevlogError. Previously that RevlogError
was converted from RuntimeError. However, D19169527 stack changed
RuntimeError to RustError, causing clindex partialmatch to raise
RustError instead, which is not handled by the Python layer.

Convert RustError to RevlogError to solve the issue.

Context:
https://fb.workplace.com/groups/scm/permalink/2591821277534143/

Reviewed By: DurhamG

Differential Revision: D19359074

fbshipit-source-id: 7b7b500cdf01272791a73ea0d94397e1397ff80b
2020-01-10 20:36:59 -08:00
Jun Wu
56b1f098dd bindings: move indexes and threading to bindings
Summary:
For some reason, our linker flag `-uinitthreading` no longer works in buck opt
build. This affects the `indexes` and `threading` Python extensions. Move them
to `bindings` to solve the problem.

Reviewed By: fanzeyi

Differential Revision: D19340538

fbshipit-source-id: 0c82b472f2916229d3c5e481954b69415b5b0dc2
2020-01-09 20:02:00 -08:00
Jun Wu
064068169b dag: rename slice to VertexName
Summary:
Per discussion, we decided to use "VertexName" as the struct name for things
like commit hashes, or the string names in tests (or the Mozilla DAG in tests).
Therefore, introduce a dedicated VertexName type and repalce all callsites to
use it.

`bytes::Bytes` is used so copying the `VertexName` is somewhat considered cheap.

This adds some overhead copying slices (and `Bytes` has some overhead). It
regresses the "building segments" benchmark from 673ms to 773ms, which seems
okay given the cleaner interface.

Reviewed By: markbt

Differential Revision: D19154905

fbshipit-source-id: 4c6d4eca67c11c10ed5f21999ccdc3f1b01695e8
2020-01-08 21:35:28 -08:00
Xavier Deguillard
83d46c7c3b pyrevisionstore: remove to_pyerr
Summary:
Python errors wrapped in a Rust Error would be incorrectly changed into a
RuntimeError instead of the original one. The map_pyerr method does the right
thing, and is also shorter, let's use it.

Reviewed By: quark-zju

Differential Revision: D19313883

fbshipit-source-id: 1ecb41a8eef9d41618905d6d00f199252d373f96
2020-01-08 12:52:43 -08:00
Mark Thomas
0822231d79 pyrenderdag: add python bindings for renderdag
Reviewed By: quark-zju

Differential Revision: D19272576

fbshipit-source-id: f79019d3506efe03675f49b9ad728582b22bd205
2020-01-06 15:00:06 -08:00
Jun Wu
9d36855838 bindings: add hgtime
Summary:
This makes it possible to use the hgtime parsing utilities in Python.
Namely, patterns like `since 6 days ago to now` can now be parsed.

Reviewed By: sfilipco

Differential Revision: D18946335

fbshipit-source-id: c12ea3437301f20a8c457525c0798f471b8f174a
2019-12-22 11:27:33 -08:00
Jun Wu
fbdecc96bb cpython-ext: move foreign error handling to pyerror
Summary:
Make cpython-ext free from business logic. This adds a bit overhead.
But it should be fine given the error cases are considered rare.

Reviewed By: markbt

Differential Revision: D19186692

fbshipit-source-id: daaffc1369a36c781d1badea822bf62a144eb54e
2019-12-22 11:27:29 -08:00
Jun Wu
d4a19306ed cpython-ext: rename fallible to anyhow_result
Summary: We no longer use the `failure` crate. Rename related methods and traits.

Reviewed By: markbt

Differential Revision: D19186694

fbshipit-source-id: 1c83a90ee12db431b7d8e09a9c2cf1d43a4c0f93
2019-12-22 11:27:29 -08:00
Jun Wu
ef3f9b932d cpython-ext: rename mod failure to error
Summary:
We no longer use the `failure` crate. Rename the module.

`cpython-failure` is dropped since it was merged into `cpython-ext`.

Reviewed By: markbt

Differential Revision: D19186693

fbshipit-source-id: 410d1491bcadd8d3272c7e2df08ecbe66fc0fef2
2019-12-22 11:27:29 -08:00
Jun Wu
2824ac99f7 cpython-ext: drop PE type parameter from map_pyerr<PE>
Summary:
The `map_pyerr<PE>` API is kind of hard to use correctly since the
code that calls `map_pyerr` does not really know the best suitable error type.

The existing code also "encourage"s the use of stdlib error types like
RuntimeError, ValueError, etc since the code is shorter, which is not great.
Error types from `mercurial.error` are better choices.

With previous diffs in this stack, we now decide the Python error type based
on the Rust error type. It's no longer necessary to specify an error type
with `map_pyerr<PE>`. So let's just drop the type parameter.

A `RustError` was added for error types that `cpython-ext` does not know how
to convert. We should avoid leaking it to the Python land by just implementing
how to convert `anyhow::Error` to `PyErr` in `cpython-ext`.

Reviewed By: markbt

Differential Revision: D19169527

fbshipit-source-id: f4563c36174cd51201b526bbc92a3f1c8a3da864
2019-12-22 11:27:28 -08:00
Jun Wu
4a4aeff284 bindings: expose IndexedLogError to Python
Summary:
Expose the indexedlog error type to Python so it can be used in `except`
clause.

Reviewed By: markbt

Differential Revision: D19169208

fbshipit-source-id: c41f08ec11f78f7714b435dbfc6a28c36b34617d
2019-12-22 11:27:27 -08:00
Jun Wu
a1d710fa58 dag: rename GroupId to Group
Summary:
Address review comment in D18640899. This makes the word `id` universally
refer to a same concept within the crate.

Reviewed By: sfilipco

Differential Revision: D18820723

fbshipit-source-id: 6803192db7e1304a72100568f8f29b90f25c7779
2019-12-20 16:14:21 -08:00
Jun Wu
fd5cb6e225 bindings: allow Python threads during indexedlog repair
Summary: This should allow us to show a spinning progress bar when repair() is working.

Reviewed By: xavierd

Differential Revision: D19056949

fbshipit-source-id: 8e058bad002bb2de29dbf8457bead0ebd1e99372
2019-12-20 16:14:18 -08:00
Stefan Filip
ef220f3f7c manifest: add the list method to the Manifest trait
Summary:
The list functionality is required by EdenFs. We want this functionality
to be well supported by the Manifest.

Reviewed By: quark-zju

Differential Revision: D18870143

fbshipit-source-id: 1ebaa713ff521226e6ace22cbd35cc841d967298
2019-12-10 10:18:19 -08:00
Stefan Filip
67b0e3a600 manifest: rename FsNode to FsNodeMetadata
Summary:
Consistency in naming. The general idea is to have Metadata types that don't
contain paths. Then we will have File, Directory and eventually FsNode that
will contain paths.

Reviewed By: quark-zju

Differential Revision: D18870141

fbshipit-source-id: a1f09add7f1c3dd4fa0348693cd3ce2fd5767fa7
2019-12-10 10:18:18 -08:00
Stefan Filip
05aa630a18 manifest-tree: rename Tree to TreeManifest
Summary:
This rename is going to make it easier to import and use outside of
manifest specific crates.

Reviewed By: quark-zju

Differential Revision: D18870142

fbshipit-source-id: 2f3ea460170308162ee834efc038b2dcedd9e233
2019-12-10 10:18:18 -08:00
Jun Wu
d754747d01 bindings: add repair to structures
Summary: Expose the repair API to the Python world.

Reviewed By: xavierd

Differential Revision: D18737913

fbshipit-source-id: c31085727589b6938c2fafb28897925aea617bc4
2019-12-09 14:15:47 -08:00
Jun Wu
7a16996ce0 revisionstore: implement indexedlog::DefaultOpenOptions
Summary:
This simplifies the logic a bit by getting a free `repair` method from
indexedlog.

Reviewed By: xavierd

Differential Revision: D18737908

fbshipit-source-id: 4988c1a83b7709b751cd1899c5663acc0c42e313
2019-12-06 19:35:04 -08:00
Stefan Filip
dd4d4a7b85 manifest: split in two crates: manifest and manifest-tree
Summary:
This isolates the core types in manifest so that it is harder to
create unsound dependencies with specific implementations.

Reviewed By: quark-zju

Differential Revision: D18843133

fbshipit-source-id: 4b866ad84d2e7d0ff2dc4ec6bd65f66548c3fe4a
2019-12-06 14:33:02 -08:00
Jun Wu
3e205e29b3 pydag: revise bindings
Summary:
The bindings APIs are out of date.

The latest idea is to write non-master commits to disk, in a different group.
There are no "segments in memory". Therefore:
- Rename `builddisk` to `build`.
- Make sure `build` updates internal states by calling `reload`.
- Drop `buildmem`.
- In case `builddisk` gets called very frequently, reduce its overhead by
  doing nothing if all commits are built already.

The constructor is revised to match the latest interface (no
"max segment level" and "set_new_segment_size" needs to be called).

Reviewed By: sfilipco

Differential Revision: D18640897

fbshipit-source-id: 76a5d0db363713dc5b763503c238c1bfa61ea02e
2019-12-05 17:01:11 -08:00
Jun Wu
75aff36c7f dag: add group support for IdMap
Summary:
This allows us to allocate Ids for non-master commits in a separate range
so they wouldn't cause fragmentation for commits in the master group.

Reviewed By: sfilipco

Differential Revision: D18640896

fbshipit-source-id: c19c53342583ded8795849584cf5f31bc86b2818
2019-12-04 22:43:23 -08:00
Jun Wu
05157a174a dag: replace u64 with the new Id type
Summary: Migrate the codebase to use the new Id type instead of an alias of u64.

Reviewed By: sfilipco

Differential Revision: D18640898

fbshipit-source-id: 415db9afeda1f8960e18c69e70f53a9eeb63cbbe
2019-12-04 22:43:23 -08:00
Stefan Filip
1f76f9e63f manifest: restructure InsertError to give back arguments
Summary:
It would be nice if the insert method would give back the path that
it failed to insert.

Reviewed By: dtolnay

Differential Revision: D18739978

fbshipit-source-id: 4c16d09750ade2f01397161129c31bcf0059a957
2019-12-02 15:53:41 -08:00
Stefan Filip
30045a27ba manifest: update python bindings to allow insertions before deletions
Summary:
The Mercurial code has the bad habbit of inserting files in what can
appear impossible locations. Sometimes files with directory names and
sometimes directories in file locations. This happens because the initial
code would do additions before deletions on Manifest implementations that
were rudimentary. As soon as we introduce validation various code paths
surface.

I tried to fix the codepaths that modify manifests but it's a losing game.
I fixed the issue that appeared in tests and a couple of issues people
reported but more situations crop up.

This is giving up on the python code.

Reviewed By: quark-zju

Differential Revision: D18737678

fbshipit-source-id: 0c97128ff67e5ba2334942b6afc404aa64a2e5f4
2019-12-02 15:53:41 -08:00
Jun Wu
353361701c metalog: add is_dirty API
Summary:
This helps test if there are uncommitted changes. It can help detecting
programming errors. For example, is metalog changed before starting a
transaction?

Reviewed By: xavierd

Differential Revision: D18538148

fbshipit-source-id: 9c82838d8174e6a93fde7108503a025660191cbf
2019-11-27 00:24:09 -08:00
David Tolnay
d1d8fb939a Switch from failure::Fail trait to std::error::Error for errors
Summary:
This diff replaces eden's dependencies on failure::Error with anyhow::Error.

Failure's error type requires all errors to have an implementation of failure's own failure::Fail trait in order for cause chains and backtraces to work. The necessary methods for this functionality have made their way into the standard library error trait, so modern error libraries build directly on std::error::Error rather than something like failure::Fail. Once we are no longer tied to failure 0.1's Fail trait, different parts of the codebase will be free to use any std::error::Error-based libraries they like while still working nicely together.

Reviewed By: xavierd

Differential Revision: D18576093

fbshipit-source-id: e2d862b659450f2969520d9b74877913fabb2e5d
2019-11-22 08:53:31 -08:00
Jun Wu
f19a760f5f blackbox: remove IndexFilter
Summary: Migrate callsites to use the new pattern matching filtering APIs.

Differential Revision: D18610738

fbshipit-source-id: 242182f4e6a58bbff5dddbb177287921c85519bc
2019-11-21 08:36:09 -08:00
Zeyi (Rice) Fan
7f0f541fa1 manifest: add FsNode to List::Directory
Reviewed By: sfilipco

Differential Revision: D18307923

fbshipit-source-id: 1844afb7b9a185f7e89c11476ee2eb34cac117c6
2019-11-19 18:01:22 -08:00
Zeyi (Rice) Fan
5b274b1a4c manifest: add tree HgId to FsNode
Summary: This diff adds `Option<HgId>` to `FsNode::Directory` because EdenFS needs the tree hash for the directories in the tree.

Reviewed By: sfilipco

Differential Revision: D18307205

fbshipit-source-id: 93db944ff1686a1c5927aa9a876a0fd008edbf9a
2019-11-19 18:01:22 -08:00
David Tolnay
066f461bcf Replace reliance on From<Context<_>> with an extension trait and constructor
Summary:
Edenapi uses a group of impls like the following as the canonical way to construct ApiError.

```
impl From<Context<ApiErrorKind>> for ApiError {
```

Downstream code would write:

```
use failure::ResultExt;

let stream_trees = config
    .get_or_default("edenapi", "streamtrees")
    .context(ApiErrorKind::BadConfig("edenapi.streamtrees"))?;
```

This relies on the way that ResultExt::context returns a unique type that is different from failure::Error.

This diff introduces a dedicated extension trait to allow the same code in the caller to continue to work even without a failure::Context<T> type, which will be required for dropping our dependency on failure::Fail and failure.

```
pub trait ApiErrorContext<T> {
    fn context(self, kind: ApiErrorKind) -> ApiResult<T>;
}
```

We also introduce a public constructor for ApiError and replace indirect construction that used to look like this:

```
error.context(ApiErrorKind::Curl).into()
```

with this instead:

```
ApiError::new(ApiErrorKind::Curl, error)
```

which is the same number of characters but clearer. The argument order matches [std::io::Error::new](https://doc.rust-lang.org/std/io/struct.Error.html#method.new).

Reviewed By: kulshrax

Differential Revision: D18574668

fbshipit-source-id: 0a56297bb942a26d75a62ca39fc16abeb4486345
2019-11-18 12:12:43 -08:00
Jun Wu
9561b5e22a treematcher: do not crash on non-utf-8 paths
Summary:
On Windows, the paths passed to the `match` functions are not encoded in utf-8
yet.  Do not crash if received them.

Reviewed By: sfilipco

Differential Revision: D18574951

fbshipit-source-id: c22ce95cb6538db0c15d60789286e2ff7d2148bc
2019-11-18 11:44:53 -08:00
David Tolnay
b1793a4416 rust: Rename Fallible<T> to Result<T>
Summary:
This diff is preparation for migrating off of failure::Fail / failure::Error for errors in favor of errors that implement std::error::Error. The Fallible terminology is unique to failure and in non-failure code we should be using Result<T>. To minimize the size of the eventual diff that removes failure, this codemod replaces all use of Fallible with Result by:

- In modules that do not use Result<T, E>, we import `failure::Fallible as Result`;
- In modules that use a mix of Result<T, E> and Fallible<T> (only 5) we define `type Result<T, E = failure::Error> = std::result::Result<T, E>` to allow both Result<T> and Result<T, E> to work simultaneously.

Reviewed By: Imxset21

Differential Revision: D18499758

fbshipit-source-id: 9f5a54c47f81fdeedbc6003cef42a1194eee55bf
2019-11-14 14:11:01 -08:00
Jun Wu
63c1b3001d treematcher: implement proper error handling
Summary:
simpkins encountered a case where the treematcher does not work with many
patterns.  It turns out `globset` has a hard-coded regex size limit (10MB).

Implement proper error handling so we can detect such issues and fallback to
slower paths.

Reviewed By: sfilipco

Differential Revision: D18500299

fbshipit-source-id: 0122ba9b0246c1536b2069a40e13261ee47f8bba
2019-11-14 11:43:18 -08:00
Adam Simpkins
ab3a7cb21f Move fb-mercurial sources into an eden/scm subdirectory.
Summary:
In preparation for merging fb-mercurial sources to the Eden repository,
move everything from the top-level directory into an `eden/scm`
subdirectory.
2019-11-13 16:04:48 -08:00