Summary:
This is merely a forward to the readlink FileInode method. This allows the
.eden directory and its symlinks to be read properly.
Reviewed By: kmancini
Differential Revision: D26619947
fbshipit-source-id: d5f8ef81ce3000c0f0a2f47007affe2c0292ef31
Summary: This adds the various types necessary to implement the READLINK RPC.
Reviewed By: kmancini
Differential Revision: D26610437
fbshipit-source-id: a64644d0499381678170da38886b5dff1c1b9270
Summary:
The LOOKUP RPC is a bit tricky as we need to special case the "." and ".."
names to represent the current and parent directories. Obtaining the parent
directory is inherently racy from the client perspective, and thus EdenFS won't
try to hold the renameLock in this case.
Reviewed By: kmancini
Differential Revision: D26597789
fbshipit-source-id: 9c8dcaf7db84f8c09f7505cab7afed48df79b754
Summary:
See D26671795. The hotfix was ineffective because repo config (and
dynamicconfig) is not loaded. This fix ensures newer versions of
hg do not depend on the hotifx.
Reviewed By: DurhamG
Differential Revision: D26679775
fbshipit-source-id: 5e78051cb7d9ac9ac6b327d895049dca2b025068
Summary:
An OnDemandUpdateDag can now track a bookmark. Every given period it will
query the changeset of the bookmark and incrementally build the dag.
Reviewed By: krallin
Differential Revision: D26656765
fbshipit-source-id: 95057863b5201f9632c654be5544922c7538f974
Summary:
For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present.
Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is.
The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed.
Reviewed By: ikostia
Differential Revision: D26618363
fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
Summary: Can save some allocations and copies by mapping the binary type to bytes::Bytes
Reviewed By: krallin
Differential Revision: D26575835
fbshipit-source-id: 518d2505e89e683d771494b53cb5fae5aea2110b
Summary:
Migrate away from streaming Thrift Python, and instead implement `eden
trace hg` in C++, which will eventually let us support Windows.
This also fixes an unexplained crash in `eden trace hg` on macOS.
Reviewed By: xavierd
Differential Revision: D25515377
fbshipit-source-id: 1916aa7f09df37e9dee3479a858932c724f8aed4
Summary:
We use an EWMA of lag values to ensure that we don't alternate between full speed writes and waiting, but instead give the DB a chance to catch up on replication after writes.
However, in S223394, we saw MyAdmin return extremely high lag values. Because our EWMA is uncapped, and we decay by half each interval, a false spike of 500 seconds takes log2(500 / 5) = 7 seconds to decay back down to the point where we will write our next chunk, assuming that lag is 0 after the false spike (not true, so can be more seconds before lag falls)
Resolve this by capping the EWMA calculation, so that we recover quickly from a false spike (max of 2 seconds with current limit), while still blocking writes while MySQL replication catches up if lag stays high.
Reviewed By: mitrandir77
Differential Revision: D26663528
fbshipit-source-id: 6e5c06f79f08ac6ebbca1b3076816a15ab8353e5
Summary: When building against Python 3.8 the MSVC compiler complains about how this macro splits inside parentheses. Not sure what changed, but looks easy enough to fix.
Reviewed By: quark-zju
Differential Revision: D26434060
fbshipit-source-id: b46059a6e676a52c99f8923b6739cffe578cc6f7
Summary: Supports Windows by adding mkscratch and scmdaemon and updating it to build python3libs.zip instead of python27.zip. Changed the name to include 'libs' on the end since the zip no longer contains Python itself.
Reviewed By: quark-zju
Differential Revision: D26381051
fbshipit-source-id: 0a2cc40df103525fdc581a4102458e82fda1f670
Summary:
In Python 3 the strings are no longer bytes, so we need to convert them before calling Windows native APIs.
I originally attempted to make all C APIs use the W() Windows APIs. This sorta worked and enabled some unicode support for Windows in Python 2, like hg status, but it meant that utf-8 path encodings were being returned and eventually passed to python functions like os.lstat() which weren't expecting utf8 encoding. I gave up and just left the Python 2 C code as is, and made a copy that uses W() APIs for Python 3. This enables unicode support on Windows where it didn't work before (at least in my testing).
Reviewed By: quark-zju
Differential Revision: D26381053
fbshipit-source-id: 69d4e18ba9fb0f3d17bad58fbcc5d0e6e61d4252
Summary:
Windows tests were failing to execute hg clone ssh://user@dummy/... lines because setconfig ui.ssh=C:/foo/bar was being translated to ui.ssh=c;c:\\foo\\bar by mingw, since it detected /foo/bar as a unix-style path. This seems to be caused by this one line reversing the slashes. I'm not sure why it exists, but deleting it makes the tests pass.
My guess is this has been broken for about a year.
Reviewed By: quark-zju
Differential Revision: D26639206
fbshipit-source-id: d89cae1ea3dd055b90ec6ee8f7cdbee2ae08b228
Summary: This diff sets up the ability for us to track requests as they are shuffled around in the `std::vector<> queue`. Since the queue is a vector, and since it is sorted everytime a new element is added or removed, we cannot keep track of elements in the queue with indices or references. Instead, we will store the requests in a shared_ptr so we can maintain a pointer to the request no matter where the request is moved around to.
Reviewed By: kmancini
Differential Revision: D26355907
fbshipit-source-id: d714d689963106a4f495221dbcfcbab758ffc7b2
Summary: The first byte is the flag.
Reviewed By: sfilipco
Differential Revision: D26654640
fbshipit-source-id: 4d66f14c73fe40a154ca7c08f9a9dff3a54ae337
Summary:
Like our other "tokio ecosystem" crates, I'm renaming this one tokio-util-02
(as in tokio-02) since this is the last version it is compatible with. Note that this had a
breaking change as well, so I updated the callsites.
Reviewed By: HarveyHunt
Differential Revision: D26636350
fbshipit-source-id: 30f7da1036c861a97717c8d26648daaae33ecfbd
Summary:
We use a forked version of Gotham in Mononoke. This isn't great, because we
have to maintain this fork. Ideally, we'd upstream our changes, but as is
they're a bit intrusive and not generally useful, which makes this hard.
I've reworked how we do our Gotham changes, and now we only need to make 1 bit
of code public, which might be easier to get upstream. Concretely, Gotham has a
concept of "connected handler" that links a Hyper request and a socket address,
but in our case we want more things. This change lets us instantiate our own
Gotham state, and then add a few more things to it as necessary.
This diff updates our code accordingly to be compatible. This also lets us trim
down on some ceremony we had to do call into Gotham
from Mononoke Server.
Reviewed By: farnz
Differential Revision: D26634653
fbshipit-source-id: 024a48ebc3f323c165ac412ef422755e8cb1c650
Summary:
When in readonly mode we can't write anything, therefore there is no need to monitor for write delay.
Also imported nonzero_ext::nonzero! macro as a small tidy up.
Reviewed By: farnz
Differential Revision: D26662396
fbshipit-source-id: 36cb2c1375a8fb72e3976290907d4c8c34ea0f4e
Summary:
Add case expressions to the templating language that allow selection of a value based on
a single determinator. The expression `case(expr, case1, then1, case2, then2, else)`
will expand to `then1` if `expr` matches `case1`, etc. If it matches none of the cases
then it expands to `else`.
This can be used to simplify long `ifeq` chains in templates.
Reviewed By: quark-zju
Differential Revision: D26631539
fbshipit-source-id: 7543e6f7baa5599c96cac75da17db73e03b918f9
Summary:
I'd like to prepare the migration to Tokio 1.0 and this is one bit of code
that needs some non-trivial changes since in Tokio 1.0, Sleep is no longer
Unpin.
Reviewed By: farnz
Differential Revision: D26610033
fbshipit-source-id: 1db4c1686fcd010e2158bcf4bb25f1e15dd19603
Summary:
Like it says in the title, this updates futures_ext to use tokio_shim, which
makes it compatible with Tokio 0.2 and 1.0.
There is one small difference in behavior here, which is that in Tokio 1.0,
sleep isn't Unpin anymore, so callers will need to call `boxed()` or use Tokio's `pin!` macro if they need
Unpin.
I do want to get as close to what upstream is doing in Tokio 1.0, so I think
it's good to keep that behavior.
Reviewed By: farnz
Differential Revision: D26610036
fbshipit-source-id: ff72275da55558fdf8fe3ad009d25cf84e108a5a
Summary: Make the auth Python bindings match the behavior of `httpconnection.readauthforuri` and omit unset fields in the returned dict.
Reviewed By: quark-zju
Differential Revision: D26439292
fbshipit-source-id: 776d8d9bd59726f5ad8d28973e9c2fdc99a2995c
Summary:
Allow the option of specifying the master tracked bookmark.
(Note: this ignores all push blocking failures!)
Reviewed By: quark-zju
Differential Revision: D26628452
fbshipit-source-id: 10dc4a13d89649355cf6c26b9be91076f2fb6cd3
Summary:
The interesting part is defaults. Particularly around various update periods.
We describe our defaults in metaconfig::types::SegmentedChangelogConfig. An
interesting example is `tailer_update_period`. By default it is
`Some(Duration::from_secs(300))`. Which means that when Segmented Changelog is
enabled we have a pause of 5 minutes before the tailers start the update
process again after an update. We want to prevent the tailer from updating a
specific repository. We do that by using an explicit config which specifies a
duration of 0 seconds. This then becomes an unset (`None`) period in
SegmentedChangelogConfig.
Same idea for all period fields. `update_to_master_bookmark_period` defaults to
`None` currently but we'll update that shortly to `Some(Duration)`.
(Note: this ignores all push blocking failures!)
Reviewed By: quark-zju
Differential Revision: D26628121
fbshipit-source-id: ccfb59b655bf9a7ca3ce2be6c03725eb9b1fe7ed
Summary:
Facebook
It looks like we have quite a few errors when people cloning fbsource on their
laptops (see https://fburl.com/odo8tii0 for more details).
"Couln't resolve host name" is one of the errors we see often, so let's retry
that. ssl errors seem to be retryable as well.
Reviewed By: ikostia
Differential Revision: D26635252
fbshipit-source-id: 2412c4f09aab3d5d45923c4b05fc350556bf9af6
Summary:
It is quite likely that this feature will be more useful if we can increase our
visibility into an entire class of source_hostnames, like mactest or
sandcastlevm, for example.
Let's add support for that.
Reviewed By: krallin
Differential Revision: D26545736
fbshipit-source-id: f84c53969674cee63880fb2cf918bcd87f852dc8
Summary:
The progress suspension is implemented in Rust. Since we're writing to the Rust
"error" and "output" streams, they should handle progress suspension directly.
This also removes the "flakiness" (progress disappears for a while when a line
is printed) when progress is shown in streampager:
lhg debugprogress -s 100 --sleep 100 --with-output --pager=on --config pager.interface=fullscreen
Reviewed By: sfilipco
Differential Revision: D26612478
fbshipit-source-id: b75bfe726c61d89495b198670f777440506fcecf
Summary:
See D23966992 (2a2971a4c7) for context. This is a continuation of D23966995 (ab88771161)
which toggles the deserialization part.
Without this change, the default tuple serialization is inefficient
in CBOR and is a disaster in Python:
In [1]: s,f=api.commithashtolocation('fbsource', repo['master'].node(), [bin('375c7ec7442d2bab635f4ae399be6604c5386eb7')])
In [2]: list(s)
Out[2]:
[{'hgid': (55, 92, 126, 199, 68, 45, 43, 171, 99, 95, 74, 227, 153, 190, 102, 4, 197, 56, 110, 183),
'location': {'descendant': (37, 71, 244, 168, 243, 219, 161, 246, 36, 253, 0, 121, 134, 114, 241, 20, 136, 105, 226, 80),
'distance': 10000}}]
With this change, the Python serialization works as expected:
In [2]: list(s)
Out[2]:
[{'hgid': '7\\~\xc7D-+\xabc_J\xe3\x99\xbef\x04\xc58n\xb7',
'location': {'descendant': '%G\xf4\xa8\xf3\xdb\xa1\xf6$\xfd\x00y\x86r\xf1\x14\x88i\xe2P',
'distance': 10000}}]
Places using HgId serialization directly has opt-in explicit serialization
format: D23966986 (0bb45fcbc4) (zstore), D23966991 (f833f03ba2) (metalog), D24009039 (9c5d20904d) (revisionstore) so
they should not be affected.
EdenAPI protocols used in production should be using `WireHgId` types so they
are not affected.
Reviewed By: sfilipco
Differential Revision: D26562685
fbshipit-source-id: 819b794272ee23a135bbed5b61706944ed0acb9f
Summary:
NFS clients will act differently depending on what kind of error is returned
from an RPC. A NFS3ERR_NOENT in response to a LOOKUP RPC may for instance
populate a negative lookup cache to avoid re-querying the same file again.
Thus, we need to translate the various errors with their corresponding
nfsstat3 values.
Reviewed By: genevievehelsel
Differential Revision: D26597788
fbshipit-source-id: d22a552196a378d6d7e9cd374e63bb8bcf436ce4
Summary:
My edenfs log is filled with the following backtrace:
Traceback (most recent call last):
File "/opt/fb/mercurial/edenscm/mercurial/commands/eden.py", line 421, in get_tree
self._fetch_tree_impl(path, manifest_node)
File "/opt/fb/mercurial/edenscm/mercurial/commands/eden.py", line 671, in _fetch_tree_impl
self.repo._prefetchtrees(path, mfnodes, [], [])
File "/opt/fb/mercurial/edenscm/hgext/treemanifest/__init__.py", line 565, in _prefetchtrees
return self._httpprefetchtrees(*args, **kwargs)
File "/opt/fb/mercurial/edenscm/hgext/treemanifest/__init__.py", line 678, in _httpprefetchtrees
self.edenapi.complete_trees(
TypeError: Expected type that converts to PySequence but received set
And indeed, the code passes a set instead of a plain list.
Reviewed By: kmancini
Differential Revision: D26545708
fbshipit-source-id: 4164154d9cc8991b596198d9324474f646cc194c
Summary:
This diff adds megarepo service methods to SCS. Each of the methods just returns "not implemented" at the moment.
The main goal of this diff is to provide a reference against which a sample client can be built.
This diffs makes two decisions that came from the source control discussion presentation:
1. expensive calls are asynchronous. This means that they are separated into an initial "request placing" call, which returns a polling token; and a polling call, which takes said token and blocks until it either times out or until the request is satisfied
1. racy calls take target location and only advance target if this location does not change until the advancement time comes. This ensures that the clients are aware that changing sync config versions or re-merging sources should be done why `megarepo_sync_changeset` calls are paused
There are also some decisions that I made while writing this, which may be questionable:
1. The polling token types are just string aliases. This is fine atm, but may be a problem in the future if we want to implement `AddScubaResponse` or `AddScubaParams` for some but not others. Moreover, now I had to implement both traits for `String`, which I am also not too happy about. Maybe I should turn polling tokens into wrapper structs right now
1. all new methods start with `megarepo_` or `poll_megarepo`. Generally I dislike name-based namespacing, but in this case the alternative is to either keep the names like `sync_changeset` (not descriptive enough), or move these methods into a separate service definition (too much overhead, and probably not desirable from operational perspective anyways). So I went with prefixes.
1. **the most questionable decision IMO** is about a config design. ATM the `SyncTargetConfig` type contains the `Target` internally. At the same time, `MegarepoChangeTargetConfigParams` contains both the `Target` and the `SyncTargetConfig` and will error out if `SyncTargetConfig` refers to a different target. The API could be made simpler by just taking `SyncTargetConfig`, but it feels a bit wrong to not have the client give us the target they want to change the config for. I can be easily persuaded to remove `Target` from `MegarepoChangeTargetConfigParams`. It can also be done incrementally, until we have any real clients, so figuring this out now is not important.
Reviewed By: markbt
Differential Revision: D26549580
fbshipit-source-id: 2d38d6273fb5ec46b846a242f4f1abf61502cb1c
Summary:
The Rust IO handles progress and streampager stuff. Switch to it so we don't
need to changing the `fout`, `ferr` when handling streampager in Python.
The chgserver logic is updated to just set raw fd 0, 1, 2 to update stdio,
since `fileno` is no longer exposed from Rust.
Manually tested the following commands, both without chg and with chg:
- lhg log -r . (no pager)
- lhg log (with streampager)
- lhg log --config pager.pager=less (with less pager)
- lhg commit (spawns pager)
- lhg debugprogress -s 100 --sleep 100 --with-output --pager=off (progress in stderr)
- lhg debugprogress -s 100 --sleep 100 --with-output --pager=on --config pager.interface=fullscreen (progress in streampager)
- lhg debugprogress -s 100 --sleep 100 --with-output --pager=on --config pager.pager='LESS= less' (progress is disabled with external pager)
Reviewed By: sfilipco
Differential Revision: D26612487
fbshipit-source-id: 8b4e36b614a0c080b93e41474f9a8fc33f890083
Summary:
Previously those fds were obtained via `fp.fileno()`. We're going to replace
the `fp` with something backed by Rust that does not have `fileno()`. Update
chg to replace the raw fds (0, 1, 2) directly. This is enough to affect both
Python and Rust's stdio so there is no need to replace `ui.fout`, `ui.ferr`
again.
Reviewed By: sfilipco
Differential Revision: D26612484
fbshipit-source-id: 5cd89e5955a1dcaad3d3132730354ee67c016bf0
Summary:
Write "\n" to flush the lines. This makes the lines visible instead of being
rewritten by the progress bar.
Reviewed By: sfilipco
Differential Revision: D26612483
fbshipit-source-id: 00cdc4af4dbed479532cc93fe183a19244c8c7c6
Summary:
Add a new API to test if a stream is a tty. This is needed to replace the
Python `fin`, `fout` etc. to Rust objects, because the Python land requires the
`istty` API. It is also useful for properly implement color detection in the
pure Rust land.
Reviewed By: sfilipco
Differential Revision: D26612480
fbshipit-source-id: 5cf79447b1d74e0031a954788db342afd48dc288
Summary:
Move IO to a separate crate so it is easier to be used in other libraries.
Practically, I'm going to add `is_tty()` API, and make some types from
`cpython-ext` implement it. If `cpython-ext` depends on `clidispatch`,
that would be too heavyweight.
Reviewed By: sfilipco
Differential Revision: D26612488
fbshipit-source-id: 00b18dadce3157a357d189aaa7d985efa6b4c80a
Summary:
Rename `pager` to `IO` matching the Rust IO APIs. This allows us to use the
Rust IO APIs from Python.
Reviewed By: sfilipco
Differential Revision: D26612485
fbshipit-source-id: d78f1c8966429e563fe7d190bbcb3bad1b0710fd
Summary:
I'm going to expose more APIs from the Rust IO such as setting progress
content, etc. Rename the module to clarify.
Reviewed By: sfilipco
Differential Revision: D26612490
fbshipit-source-id: 136ea27a733b09557d02077e68ce51184125ade1
Summary:
Unlike stderr or stdout, writing to progress clears the previous progress text.
Update the function name to clarify.
Reviewed By: sfilipco
Differential Revision: D26612481
fbshipit-source-id: 382182fdcc713f92117d01f53b8da47d195b06cb
Summary:
Update `write_progress` to write the progress to stderr if streampager is not
used.
Reviewed By: sfilipco
Differential Revision: D26612482
fbshipit-source-id: 3034252765024df2a6bda7347eff35f07f3186cb
Summary:
streampager uses "\f" to indicate "the end of progress". Move that logic from
Python to Rust. This hides the streampager implementation detail and allows
us to implement `write_progress` for non-streampager cases.
Reviewed By: sfilipco
Differential Revision: D26612489
fbshipit-source-id: 7f41c3af78a0c098de510f86c86726c43931cb6d
Summary:
There seems no need for the progress bar text to be binary (it is not used to
redirect to binary files). Let's change the type to make future changes easier.
Reviewed By: sfilipco
Differential Revision: D26612479
fbshipit-source-id: 1da7950d5ab2b590963f1591e2f1dd5ae8e0116c
