Summary:
Add a few additional type annotations on some functions.
This did require some minor code restructuring in `filestore.setfile()` to
work around pyre's unwillingness to unwrap `Optional` member variables.
Reviewed By: quark-zju
Differential Revision: D19715243
fbshipit-source-id: 8078329c1e4a50ad0aa6c765d42a89b5ed58e7bf
Summary:
Add more type annotations to vfs.py and some of the platform-specific
functions that it calls.
This did catch a couple of cases where string/bytes conversion were not being
done properly in Python 3.
Reviewed By: quark-zju
Differential Revision: D19678717
fbshipit-source-id: b3979cc5e0668ddbb93372dee25ef02ea1867d6f
Summary:
Fix various type issues when running chg under Python 3.
Enable chg in setup3.py build.
This should make tests run faster. For example, test-rebase-detach.t
now completes in 8 seconds, down from 29 seconds.
Reviewed By: xavierd
Differential Revision: D19702535
fbshipit-source-id: 8928b1b920b9b52fd03dc86f996da18f2405f146
Summary:
Drop stdoutbytes/stdinbytes. They make things unnecessarily complicated
(especially for chg / Rust dispatch entry point).
The new idea is IO are using bytes. Text are written in utf-8 (Python 3) or
local encoding (Python 2). To make stdout behave reasonably on systems not
using utf-8 locale (ex. Windows), we might add a Rust binding to Rust's stdout,
which does the right thing:
- When writing to stdout console, expect text to be utf-8 encoded and do proper decoding.
- Wehn writing to stdout file, write the raw bytes without translation.
Note Python's `sys.stdout.buffer` does not do translation when writing to stdout console
like Rust's stdout.
For now, my main motivation of this change is to fix chg on Python 3.
Reviewed By: xavierd
Differential Revision: D19702533
fbshipit-source-id: 74704c83e1b200ff66fb3a2d23d97ff21c7239c8
Summary:
My earlier refactor moved a watchmanclient.getcurrentclock() call out
of a "try/except Exception" block. This meant any watchman errors broke the hg
status entirely, instead of falling back to the old status path.
Let's add the logic around this watchman path.
Reviewed By: sfilipco
Differential Revision: D19709391
fbshipit-source-id: bfb0221ba405a926babddaf73c8e70924e4d9b25
Summary:
Implementation based on what we do in bcm layer
for reporting (non histogram) port stat counters. Abstracting
this lib out so we can reuse this code in both SAI and bcm
layers. Plus pulling this out of BcmPort makes it much
easier to unit test this code.
Reviewed By: boryas
Differential Revision: D19676641
fbshipit-source-id: 88aa20f20339729f5387500bcf6084871f4e7caa
Summary:
The "repo" type is a bit hard to type since it might be wrapped by extensions,
and the decorators like "unfilteredmethod" are too hard for type checkers.
Provide a stub for it so typecheck can work. The stub was generated by `pytype`,
I made a small change to `close` to make pyre happy.
Reviewed By: DurhamG
Differential Revision: D19677410
fbshipit-source-id: 71afb3fe78a75ee269c8005d18eee3a807b50df6
Summary:
This commit enables mounting and unmounting using the
new helper, and fixes up the behavior of `eden redirect list`
and `eden gc` so that they have reasonable behavior when APFS
is in the mix.
The semantics of the `TARGET` field in the redirect list output
become a bit hazy when using APFS volumes; it's a bit less important
because the data isn't effectively moved to a different part of
the filesystem namespace: we're just mounting a different filesystem
over the top and it can only be accessed through its mountpoint.
Reviewed By: chadaustin
Differential Revision: D19403162
fbshipit-source-id: 136a7b02f605cde0a74016608179924be801eda9
Summary:
We've had a small proportion of our users run into problems
with hdiutil and diskimages-helper, where those components get into
an unhappy state and effectively block operating on the redirections.
This diff introduces a new utility that is intended to replace the
use of disk image files with APFS volumes that we mount in the
appropriate places.
The intention is that we will teach `eden redirect` to use this tool
when available, rather than disk images.
macOS's security model is weird: it is perfectly valid for a non-privileged
user to create and delete APFS volumes in the APFS storage container,
but root privs are required to mount it into the VFS.
The intent is that we deploy this utility setuid root to minimize
the fan out--this way we won't need to teach the priv helper about
this kind of redirection.
There are a couple of subcommands demonstrated in the test plan.
Reviewed By: chadaustin
Differential Revision: D19323850
fbshipit-source-id: 35556f841e49e5c4b77679b756af9093222f4500
Summary:
To avoid accidents when people run `eden rm` with a subdirectory of
the mount, require that it be given the checkout root (or a symlink to
it).
Reviewed By: genevievehelsel
Differential Revision: D19549912
fbshipit-source-id: ae0a43e9164f73f5ad9c2f40f1b7226a1e85a567
Summary:
Add type hints for `vfs.join()`. Not all subclasses supported joining
multiple arguments, so fix them all to actually implement the full API.
Reviewed By: farnz
Differential Revision: D19678712
fbshipit-source-id: f7f77eee416cd5edda756a8423cbe3630e195d4a
Summary:
Update setup.py to exclude `edenscm/mercurial/pycompat3.py` when building
for Python 2.
Reviewed By: quark-zju
Differential Revision: D19703778
fbshipit-source-id: 269f1d4e746eb5112a9e577f43c040f2946b07d1
Summary:
I was supposed to do it D19691216, but accidentally reverted all the changes :(
This diff brings them back
Reviewed By: farnz
Differential Revision: D19698315
fbshipit-source-id: 5f20d27dfd33f3464d126ad72e6d49485366be3d
Summary:
There were some unsafe uses of folly::collect, especially in
the diff code. Switch to collectSafe.
Reviewed By: simpkins
Differential Revision: D19529784
fbshipit-source-id: e0ac920a2d2b44767a453f2c41a9166e72246ab6
Summary:
folly::collect's Future is completed immediately when any of its input
future is completed with an exception. This makes forking off subtasks
and then joining with folly::collect dangerous when closures don't
hold pointers or references to parameters, including `this`. Introduce
a collectSafe function, which has the same signature as folly::collect
but unconditionally waits until all futures are completed.
Reviewed By: simpkins
Differential Revision: D19529772
fbshipit-source-id: b3b493fcbb0d9058d2d6e5c8a064dfdae3a43e57
Summary: Matchers were mixing str and bytes - use decodeutf8 on the place using bytes, and enable all the tests that now pass.
Reviewed By: quark-zju
Differential Revision: D19679076
fbshipit-source-id: 2b90cfbf690b2365a1302efd9db72347a295c977
Summary: This gives type checker some hints. Pyre can now look into more areas.
Reviewed By: markbt
Differential Revision: D19676085
fbshipit-source-id: db05b4238051953eb7f8af44da220825ec589c3c
Summary:
This gives type checker some hints. Pyre was able to find some issues,
which are fixed.
Reviewed By: markbt
Differential Revision: D19674436
fbshipit-source-id: 44d6df6037f27e2edeff4d343a9fdec5e51420b3
Summary: This allows type checkers to understand the i18n._ calls.
Reviewed By: markbt
Differential Revision: D19673677
fbshipit-source-id: be2cc72f70e56925007485c786f0d72b5cd40a1a
Summary: Dependencies are getting quite complicated to handle in the generator with all the corner cases. This refactory should make it easier to work with them.
Reviewed By: ikostia
Differential Revision: D19690523
fbshipit-source-id: 64c0289679e492db03837ebb02d01207a1548665
Summary: `BlobRepo::rechunk_file_by_content_id` only used once and just calls `filestore::rechunk`, so it is safe to remove it
Reviewed By: StanislavGlebik
Differential Revision: D19691267
fbshipit-source-id: f40b4dc14295f91f6134c29ddbfeaa8abd66535e
Summary:
This is intended to be used in implementation of 'identify' API of
SCS.
Reviewed By: markbt
Differential Revision: D19640581
fbshipit-source-id: b93c4d8b60b59ce67b5090394083707e6ee8a3c7
Summary:
Now that we have FilenodesOnlyPublic derived data we no longer need to generate
filenods when we generate hg changesets.
This is important, because we want to generate hg changesets for both public
and draft commits, but we want to generate filenodes only for public commits.
By separating filenodes generation from hg changeset generation we can finally
achieve that.
This diff is the final step - it removes IncompleteFilenodes struct completely.
A few caveats:
1) This diff also removes regenerate_filenodes binary - backfill_derived_data can
be used instead.
2) In D18748102 we added a hack that force-backfilled filenodes when a merge
commit is pushed. This hack is no longer necessary and it was removed, but we
still want to protect against pushing too many commits with no filenodes
generated for them. So instead of force backfilling the filenodes the code now
just checks how commits are missing filenodes.
Reviewed By: aslpavel
Differential Revision: D19683369
fbshipit-source-id: adf613defa50400e55a95dbe14053d954b88016b
Summary:
The idea of the test was that this should be a stack of empty commits, and deriving
the top one also derives the bottom commit. The test happened to pass because
we still generate filenodes during hg changeset generation, however it started
to break in the next diff. So this diff fixes the test.
While at it also fix the comment
Reviewed By: farnz
Differential Revision: D19690293
fbshipit-source-id: d46f5686c8468b5a180104c9e5313e51eafc3092
Summary:
This function can be used to count how many ancestors of a commit do not have
data derived. It will be used in the next diff
Note that count_underived() function traverses the commit graph, so it can be
quite expensive. That's why `limit` parameter is required to bound the walk.
Reviewed By: farnz
Differential Revision: D19683368
fbshipit-source-id: 2ca07db19fc10b5ea86107b084a28072ec4703fa
Summary: Phases::is_public was only used from test code, so moved it to be test only.
Reviewed By: StanislavGlebik
Differential Revision: D19644697
fbshipit-source-id: 1d770a9e40fa2593ebe6f34b80ead90495e69d24
Summary:
The is_public check calls phases.get_public_derive, which in turn calls repo.get_bonsai_heads_maybe_stale. The later is a race as the heads may be out of date vs the changesets being walked if the walker is very near a recent commit.
This change loads the heads information once, so that it can be used in calls to get_public_derive throughout the walk
Also removes the SqlPhases dependency, walker now works using dyn Phases
Reviewed By: krallin
Differential Revision: D19601580
fbshipit-source-id: 31779c4d858cb4fa3ca98dc715d624a6b2ad8cd6
Summary:
Updates the Phases trait and its cached implementation so that the ephmeral_derive option is visible.
This api change is in preparation for decoupling the by the walker from SqlPhases.
Reviewed By: krallin
Differential Revision: D19621212
fbshipit-source-id: c09006e8d05a7d49e9c4ba2c3cc240caa216f023
Summary:
We have two ways to generate hg changesets in Mononoke: generate new hg
changesets from a bonsai changeset (see
BlobRepo::get_hg_from_bonsai_changeset() function) or accept a commit
generated by Mercurial either via blobimport or through the wireproto unbundle
command(CreateChangeset).
The main goal of the stack is to remove filenode generation from both of those
places, and this diff cleans up the CreateChangeset. By doing it we are able to get rid
of an ugly hack of specifying whether a commit is draft or not.
Reviewed By: krallin
Differential Revision: D19516956
fbshipit-source-id: 77921612a919a52f6ce1ddfbf55944ca300db46e
Summary:
Begin adding type hints to many of the vfs methods.
These type hints indicate that the vfs objects are always opened in binary
mode. While it is possible to open a vfs in text mode if you explicitly
specify the argument `text=True`, this is very rare. If we run into cases
in the future that do need type hints to indicate TextIO mode I think it would
be better to split those out into completely separate functions, rather than
having functions that sometimes return bytes and sometimes return str based on
the arguments used.
Reviewed By: markbt
Differential Revision: D19673538
fbshipit-source-id: 8683223f28964d4b43fe131d4f16f8877dcbe777
Summary:
Make `atomictempfile` derive from `typing.BinaryIO` so that the type checker
understands it can be used like a file.
This required defining all `BinaryIO` methods on this class. This does result
in a fair amount of additional boilerplate code, but seems like the easiest
approach for now.
Reviewed By: markbt
Differential Revision: D19673539
fbshipit-source-id: b5775ca79b3af456b45e4ef5480711d7095e7949
