Summary:
I accidentally broke this in D26544410 (097e4ad00c) when I updated it to use
schedule_stats_aggregation_preview (in Tokio 0.2 and up, you can't create
an interval stream off the runtime, but in Tokio 0.1 you could).
We only use this method in 2 places, so it probably makes sense to just get rid
of it anyway, which is what this diff does. The alternative is better as it
spawns this unconditionally, so if we get it wrong, it'll fail in tests,
even though our tests don't pass `--fb303-port`, whereas
`start_fb303_and_stats_agg` will only start stats aggregation if its passed.
Reviewed By: ahornby
Differential Revision: D26690223
fbshipit-source-id: 7d151a3c46fa428f00ac32601da161609fb498f7
Summary:
The NFS RPC description is filled with variant over nfsstat3 with only 2 cases:
NFS3_OK, and the default one. I was feeling bad having to write the same code
again and again for the deserialize part of the XdrTrait, and thus came up with
a template solution that enables us to remove all of the boilerplate code.
The only drawback is that this moves the variant one additional layer down, and
that means an extra layer of brace when initializing it. From what I know,
this is caused by the -Wmissing-braces warning, maybe we could remove that
warning?
Reviewed By: kmancini
Differential Revision: D26627163
fbshipit-source-id: ab56dee42273f180b2edf4f529221a15154ac2fb
Summary: This seems to have been reverted by accident in D26618363 (f317302b0f).
Differential Revision: D26689734
fbshipit-source-id: e86451716cab3cc62f517c3f5fca7898d1a25095
Summary:
"value not present in store" is an unhelpful error message once
serialized through Thrift etc. Provide more useful context when a key
is missing.
Reviewed By: fanzeyi
Differential Revision: D26678102
fbshipit-source-id: 514ac2fe580d1dd7c67fc20c89b75e5d8121c329
Summary:
We got a report about windows in vscode being significantly slower than on
macOs/Linux (https://fburl.com/nv9xwc09) to the point that it can't be
explained by "Windows is just slower". As the post suggests we had a suspicion
it might be related to an issue described in T33479254.
After some debugging I can confirm the following:
1) I can reliably (but not 100% of the time) reproduce the issue on my windows
laptop. I just need to modify a file and do "Quick commit" or "Quick amend" from vscode,
and note that command finished quite fast (just a few seconds, I verified it by
running "hg st" and checking that working copy is now clean, and by running "hg
log -r ." and checking that hash has changed), but vscode takes longer to
notice that (i.e. it keeps spinning and showing that amend is still running).
2) By staring at "Process explorer" I noticed that the problem seem to be in
"hg.exe cloud backup" i.e. when "cloud backup" finishes then vscode stops
spinning. So I suspected it to be a problem.
3) As the next step I started to suspend the process in "Process Explorer"
(note that I started it as admin and SYSTEM user, whatever it means). Then I
looked through file handles in Process explorer and was closing them 1 by 1.
Evetually I noticed that closing handles for named pipes like
"\Device\NamedPipe\uv\0000013E91115170-26220" make vscode stop spinning (it was
usually necessary to close it in hg.exe cloud backup process and all child processes as
well).
4) I also looked at handle value, and noticed that it's bigger than 2048 (0x848).
So currently my suspicion is that the problem is that we don't close enought
handles, and this diff bumps it as a temporary workaround and also walk over handles that multiple of 4 only. quark-zju has an
upstream improvement that would make this hack go away
(https://github.com/rust-lang/rust/pull/75551/commits) but it's not landed yet.
So for now let's try to bump the magic number.
Reviewed By: quark-zju
Differential Revision: D26668773
fbshipit-source-id: ed1c203260a52c3e5449b7b06cf4ecbe4dcf6477
Summary:
Previously, when an argument was just a plan filehandle, I just bypassed adding
the structure and just read the handle directly from the stream. kmancini
correctly pointed out that this is a bit confusing, thus let's add the arg
struct everywhere to cut that confusion.
Reviewed By: kmancini
Differential Revision: D26649568
fbshipit-source-id: bc4d6e519450f228d151a3de7acdfc1002c43b38
Summary:
This is merely a forward to the readlink FileInode method. This allows the
.eden directory and its symlinks to be read properly.
Reviewed By: kmancini
Differential Revision: D26619947
fbshipit-source-id: d5f8ef81ce3000c0f0a2f47007affe2c0292ef31
Summary: This adds the various types necessary to implement the READLINK RPC.
Reviewed By: kmancini
Differential Revision: D26610437
fbshipit-source-id: a64644d0499381678170da38886b5dff1c1b9270
Summary:
The LOOKUP RPC is a bit tricky as we need to special case the "." and ".."
names to represent the current and parent directories. Obtaining the parent
directory is inherently racy from the client perspective, and thus EdenFS won't
try to hold the renameLock in this case.
Reviewed By: kmancini
Differential Revision: D26597789
fbshipit-source-id: 9c8dcaf7db84f8c09f7505cab7afed48df79b754
Summary:
See D26671795. The hotfix was ineffective because repo config (and
dynamicconfig) is not loaded. This fix ensures newer versions of
hg do not depend on the hotifx.
Reviewed By: DurhamG
Differential Revision: D26679775
fbshipit-source-id: 5e78051cb7d9ac9ac6b327d895049dca2b025068
Summary:
An OnDemandUpdateDag can now track a bookmark. Every given period it will
query the changeset of the bookmark and incrementally build the dag.
Reviewed By: krallin
Differential Revision: D26656765
fbshipit-source-id: 95057863b5201f9632c654be5544922c7538f974
Summary:
For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present.
Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is.
The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed.
Reviewed By: ikostia
Differential Revision: D26618363
fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
Summary: Can save some allocations and copies by mapping the binary type to bytes::Bytes
Reviewed By: krallin
Differential Revision: D26575835
fbshipit-source-id: 518d2505e89e683d771494b53cb5fae5aea2110b
Summary:
Migrate away from streaming Thrift Python, and instead implement `eden
trace hg` in C++, which will eventually let us support Windows.
This also fixes an unexplained crash in `eden trace hg` on macOS.
Reviewed By: xavierd
Differential Revision: D25515377
fbshipit-source-id: 1916aa7f09df37e9dee3479a858932c724f8aed4
Summary:
We use an EWMA of lag values to ensure that we don't alternate between full speed writes and waiting, but instead give the DB a chance to catch up on replication after writes.
However, in S223394, we saw MyAdmin return extremely high lag values. Because our EWMA is uncapped, and we decay by half each interval, a false spike of 500 seconds takes log2(500 / 5) = 7 seconds to decay back down to the point where we will write our next chunk, assuming that lag is 0 after the false spike (not true, so can be more seconds before lag falls)
Resolve this by capping the EWMA calculation, so that we recover quickly from a false spike (max of 2 seconds with current limit), while still blocking writes while MySQL replication catches up if lag stays high.
Reviewed By: mitrandir77
Differential Revision: D26663528
fbshipit-source-id: 6e5c06f79f08ac6ebbca1b3076816a15ab8353e5
Summary: When building against Python 3.8 the MSVC compiler complains about how this macro splits inside parentheses. Not sure what changed, but looks easy enough to fix.
Reviewed By: quark-zju
Differential Revision: D26434060
fbshipit-source-id: b46059a6e676a52c99f8923b6739cffe578cc6f7
Summary: Supports Windows by adding mkscratch and scmdaemon and updating it to build python3libs.zip instead of python27.zip. Changed the name to include 'libs' on the end since the zip no longer contains Python itself.
Reviewed By: quark-zju
Differential Revision: D26381051
fbshipit-source-id: 0a2cc40df103525fdc581a4102458e82fda1f670
Summary:
In Python 3 the strings are no longer bytes, so we need to convert them before calling Windows native APIs.
I originally attempted to make all C APIs use the W() Windows APIs. This sorta worked and enabled some unicode support for Windows in Python 2, like hg status, but it meant that utf-8 path encodings were being returned and eventually passed to python functions like os.lstat() which weren't expecting utf8 encoding. I gave up and just left the Python 2 C code as is, and made a copy that uses W() APIs for Python 3. This enables unicode support on Windows where it didn't work before (at least in my testing).
Reviewed By: quark-zju
Differential Revision: D26381053
fbshipit-source-id: 69d4e18ba9fb0f3d17bad58fbcc5d0e6e61d4252
Summary:
Windows tests were failing to execute hg clone ssh://user@dummy/... lines because setconfig ui.ssh=C:/foo/bar was being translated to ui.ssh=c;c:\\foo\\bar by mingw, since it detected /foo/bar as a unix-style path. This seems to be caused by this one line reversing the slashes. I'm not sure why it exists, but deleting it makes the tests pass.
My guess is this has been broken for about a year.
Reviewed By: quark-zju
Differential Revision: D26639206
fbshipit-source-id: d89cae1ea3dd055b90ec6ee8f7cdbee2ae08b228
Summary: This diff sets up the ability for us to track requests as they are shuffled around in the `std::vector<> queue`. Since the queue is a vector, and since it is sorted everytime a new element is added or removed, we cannot keep track of elements in the queue with indices or references. Instead, we will store the requests in a shared_ptr so we can maintain a pointer to the request no matter where the request is moved around to.
Reviewed By: kmancini
Differential Revision: D26355907
fbshipit-source-id: d714d689963106a4f495221dbcfcbab758ffc7b2
Summary: The first byte is the flag.
Reviewed By: sfilipco
Differential Revision: D26654640
fbshipit-source-id: 4d66f14c73fe40a154ca7c08f9a9dff3a54ae337
Summary:
Like our other "tokio ecosystem" crates, I'm renaming this one tokio-util-02
(as in tokio-02) since this is the last version it is compatible with. Note that this had a
breaking change as well, so I updated the callsites.
Reviewed By: HarveyHunt
Differential Revision: D26636350
fbshipit-source-id: 30f7da1036c861a97717c8d26648daaae33ecfbd
Summary:
We use a forked version of Gotham in Mononoke. This isn't great, because we
have to maintain this fork. Ideally, we'd upstream our changes, but as is
they're a bit intrusive and not generally useful, which makes this hard.
I've reworked how we do our Gotham changes, and now we only need to make 1 bit
of code public, which might be easier to get upstream. Concretely, Gotham has a
concept of "connected handler" that links a Hyper request and a socket address,
but in our case we want more things. This change lets us instantiate our own
Gotham state, and then add a few more things to it as necessary.
This diff updates our code accordingly to be compatible. This also lets us trim
down on some ceremony we had to do call into Gotham
from Mononoke Server.
Reviewed By: farnz
Differential Revision: D26634653
fbshipit-source-id: 024a48ebc3f323c165ac412ef422755e8cb1c650
Summary:
When in readonly mode we can't write anything, therefore there is no need to monitor for write delay.
Also imported nonzero_ext::nonzero! macro as a small tidy up.
Reviewed By: farnz
Differential Revision: D26662396
fbshipit-source-id: 36cb2c1375a8fb72e3976290907d4c8c34ea0f4e
Summary:
Add case expressions to the templating language that allow selection of a value based on
a single determinator. The expression `case(expr, case1, then1, case2, then2, else)`
will expand to `then1` if `expr` matches `case1`, etc. If it matches none of the cases
then it expands to `else`.
This can be used to simplify long `ifeq` chains in templates.
Reviewed By: quark-zju
Differential Revision: D26631539
fbshipit-source-id: 7543e6f7baa5599c96cac75da17db73e03b918f9
Summary:
I'd like to prepare the migration to Tokio 1.0 and this is one bit of code
that needs some non-trivial changes since in Tokio 1.0, Sleep is no longer
Unpin.
Reviewed By: farnz
Differential Revision: D26610033
fbshipit-source-id: 1db4c1686fcd010e2158bcf4bb25f1e15dd19603
Summary:
Like it says in the title, this updates futures_ext to use tokio_shim, which
makes it compatible with Tokio 0.2 and 1.0.
There is one small difference in behavior here, which is that in Tokio 1.0,
sleep isn't Unpin anymore, so callers will need to call `boxed()` or use Tokio's `pin!` macro if they need
Unpin.
I do want to get as close to what upstream is doing in Tokio 1.0, so I think
it's good to keep that behavior.
Reviewed By: farnz
Differential Revision: D26610036
fbshipit-source-id: ff72275da55558fdf8fe3ad009d25cf84e108a5a
Summary: Make the auth Python bindings match the behavior of `httpconnection.readauthforuri` and omit unset fields in the returned dict.
Reviewed By: quark-zju
Differential Revision: D26439292
fbshipit-source-id: 776d8d9bd59726f5ad8d28973e9c2fdc99a2995c
Summary:
Allow the option of specifying the master tracked bookmark.
(Note: this ignores all push blocking failures!)
Reviewed By: quark-zju
Differential Revision: D26628452
fbshipit-source-id: 10dc4a13d89649355cf6c26b9be91076f2fb6cd3
Summary:
The interesting part is defaults. Particularly around various update periods.
We describe our defaults in metaconfig::types::SegmentedChangelogConfig. An
interesting example is `tailer_update_period`. By default it is
`Some(Duration::from_secs(300))`. Which means that when Segmented Changelog is
enabled we have a pause of 5 minutes before the tailers start the update
process again after an update. We want to prevent the tailer from updating a
specific repository. We do that by using an explicit config which specifies a
duration of 0 seconds. This then becomes an unset (`None`) period in
SegmentedChangelogConfig.
Same idea for all period fields. `update_to_master_bookmark_period` defaults to
`None` currently but we'll update that shortly to `Some(Duration)`.
(Note: this ignores all push blocking failures!)
Reviewed By: quark-zju
Differential Revision: D26628121
fbshipit-source-id: ccfb59b655bf9a7ca3ce2be6c03725eb9b1fe7ed
Summary:
Facebook
It looks like we have quite a few errors when people cloning fbsource on their
laptops (see https://fburl.com/odo8tii0 for more details).
"Couln't resolve host name" is one of the errors we see often, so let's retry
that. ssl errors seem to be retryable as well.
Reviewed By: ikostia
Differential Revision: D26635252
fbshipit-source-id: 2412c4f09aab3d5d45923c4b05fc350556bf9af6
Summary:
It is quite likely that this feature will be more useful if we can increase our
visibility into an entire class of source_hostnames, like mactest or
sandcastlevm, for example.
Let's add support for that.
Reviewed By: krallin
Differential Revision: D26545736
fbshipit-source-id: f84c53969674cee63880fb2cf918bcd87f852dc8
Summary:
The progress suspension is implemented in Rust. Since we're writing to the Rust
"error" and "output" streams, they should handle progress suspension directly.
This also removes the "flakiness" (progress disappears for a while when a line
is printed) when progress is shown in streampager:
lhg debugprogress -s 100 --sleep 100 --with-output --pager=on --config pager.interface=fullscreen
Reviewed By: sfilipco
Differential Revision: D26612478
fbshipit-source-id: b75bfe726c61d89495b198670f777440506fcecf
Summary:
See D23966992 (2a2971a4c7) for context. This is a continuation of D23966995 (ab88771161)
which toggles the deserialization part.
Without this change, the default tuple serialization is inefficient
in CBOR and is a disaster in Python:
In [1]: s,f=api.commithashtolocation('fbsource', repo['master'].node(), [bin('375c7ec7442d2bab635f4ae399be6604c5386eb7')])
In [2]: list(s)
Out[2]:
[{'hgid': (55, 92, 126, 199, 68, 45, 43, 171, 99, 95, 74, 227, 153, 190, 102, 4, 197, 56, 110, 183),
'location': {'descendant': (37, 71, 244, 168, 243, 219, 161, 246, 36, 253, 0, 121, 134, 114, 241, 20, 136, 105, 226, 80),
'distance': 10000}}]
With this change, the Python serialization works as expected:
In [2]: list(s)
Out[2]:
[{'hgid': '7\\~\xc7D-+\xabc_J\xe3\x99\xbef\x04\xc58n\xb7',
'location': {'descendant': '%G\xf4\xa8\xf3\xdb\xa1\xf6$\xfd\x00y\x86r\xf1\x14\x88i\xe2P',
'distance': 10000}}]
Places using HgId serialization directly has opt-in explicit serialization
format: D23966986 (0bb45fcbc4) (zstore), D23966991 (f833f03ba2) (metalog), D24009039 (9c5d20904d) (revisionstore) so
they should not be affected.
EdenAPI protocols used in production should be using `WireHgId` types so they
are not affected.
Reviewed By: sfilipco
Differential Revision: D26562685
fbshipit-source-id: 819b794272ee23a135bbed5b61706944ed0acb9f
Summary:
NFS clients will act differently depending on what kind of error is returned
from an RPC. A NFS3ERR_NOENT in response to a LOOKUP RPC may for instance
populate a negative lookup cache to avoid re-querying the same file again.
Thus, we need to translate the various errors with their corresponding
nfsstat3 values.
Reviewed By: genevievehelsel
Differential Revision: D26597788
fbshipit-source-id: d22a552196a378d6d7e9cd374e63bb8bcf436ce4
Summary:
My edenfs log is filled with the following backtrace:
Traceback (most recent call last):
File "/opt/fb/mercurial/edenscm/mercurial/commands/eden.py", line 421, in get_tree
self._fetch_tree_impl(path, manifest_node)
File "/opt/fb/mercurial/edenscm/mercurial/commands/eden.py", line 671, in _fetch_tree_impl
self.repo._prefetchtrees(path, mfnodes, [], [])
File "/opt/fb/mercurial/edenscm/hgext/treemanifest/__init__.py", line 565, in _prefetchtrees
return self._httpprefetchtrees(*args, **kwargs)
File "/opt/fb/mercurial/edenscm/hgext/treemanifest/__init__.py", line 678, in _httpprefetchtrees
self.edenapi.complete_trees(
TypeError: Expected type that converts to PySequence but received set
And indeed, the code passes a set instead of a plain list.
Reviewed By: kmancini
Differential Revision: D26545708
fbshipit-source-id: 4164154d9cc8991b596198d9324474f646cc194c
Summary:
This diff adds megarepo service methods to SCS. Each of the methods just returns "not implemented" at the moment.
The main goal of this diff is to provide a reference against which a sample client can be built.
This diffs makes two decisions that came from the source control discussion presentation:
1. expensive calls are asynchronous. This means that they are separated into an initial "request placing" call, which returns a polling token; and a polling call, which takes said token and blocks until it either times out or until the request is satisfied
1. racy calls take target location and only advance target if this location does not change until the advancement time comes. This ensures that the clients are aware that changing sync config versions or re-merging sources should be done why `megarepo_sync_changeset` calls are paused
There are also some decisions that I made while writing this, which may be questionable:
1. The polling token types are just string aliases. This is fine atm, but may be a problem in the future if we want to implement `AddScubaResponse` or `AddScubaParams` for some but not others. Moreover, now I had to implement both traits for `String`, which I am also not too happy about. Maybe I should turn polling tokens into wrapper structs right now
1. all new methods start with `megarepo_` or `poll_megarepo`. Generally I dislike name-based namespacing, but in this case the alternative is to either keep the names like `sync_changeset` (not descriptive enough), or move these methods into a separate service definition (too much overhead, and probably not desirable from operational perspective anyways). So I went with prefixes.
1. **the most questionable decision IMO** is about a config design. ATM the `SyncTargetConfig` type contains the `Target` internally. At the same time, `MegarepoChangeTargetConfigParams` contains both the `Target` and the `SyncTargetConfig` and will error out if `SyncTargetConfig` refers to a different target. The API could be made simpler by just taking `SyncTargetConfig`, but it feels a bit wrong to not have the client give us the target they want to change the config for. I can be easily persuaded to remove `Target` from `MegarepoChangeTargetConfigParams`. It can also be done incrementally, until we have any real clients, so figuring this out now is not important.
Reviewed By: markbt
Differential Revision: D26549580
fbshipit-source-id: 2d38d6273fb5ec46b846a242f4f1abf61502cb1c
Summary:
The Rust IO handles progress and streampager stuff. Switch to it so we don't
need to changing the `fout`, `ferr` when handling streampager in Python.
The chgserver logic is updated to just set raw fd 0, 1, 2 to update stdio,
since `fileno` is no longer exposed from Rust.
Manually tested the following commands, both without chg and with chg:
- lhg log -r . (no pager)
- lhg log (with streampager)
- lhg log --config pager.pager=less (with less pager)
- lhg commit (spawns pager)
- lhg debugprogress -s 100 --sleep 100 --with-output --pager=off (progress in stderr)
- lhg debugprogress -s 100 --sleep 100 --with-output --pager=on --config pager.interface=fullscreen (progress in streampager)
- lhg debugprogress -s 100 --sleep 100 --with-output --pager=on --config pager.pager='LESS= less' (progress is disabled with external pager)
Reviewed By: sfilipco
Differential Revision: D26612487
fbshipit-source-id: 8b4e36b614a0c080b93e41474f9a8fc33f890083
Summary:
Previously those fds were obtained via `fp.fileno()`. We're going to replace
the `fp` with something backed by Rust that does not have `fileno()`. Update
chg to replace the raw fds (0, 1, 2) directly. This is enough to affect both
Python and Rust's stdio so there is no need to replace `ui.fout`, `ui.ferr`
again.
Reviewed By: sfilipco
Differential Revision: D26612484
fbshipit-source-id: 5cd89e5955a1dcaad3d3132730354ee67c016bf0
Summary:
Write "\n" to flush the lines. This makes the lines visible instead of being
rewritten by the progress bar.
Reviewed By: sfilipco
Differential Revision: D26612483
fbshipit-source-id: 00cdc4af4dbed479532cc93fe183a19244c8c7c6