Summary: We have a number of sleeps in our integration tests. The two main reasons are configs & tunables that need reloading. Currently, we have no way of force-reloading those.
Reviewed By: krallin
Differential Revision: D26615732
fbshipit-source-id: 217c4ae039abd398972b4a9764d08e18d6182493
Summary:
This dag periodically reloads the dag from storage.
It currently loads a simple dag that has no update logic because that is what
the manager returs. It's not relevant for this code.
This is probably the last piece before we refactor construction to take a
SegmentedChangelogConfig. To be seen how much will be strict types and how much
will be Arc<dyn SegmentedChangelog>.
Reviewed By: krallin
Differential Revision: D26681458
fbshipit-source-id: 6056d00db6f25616e8158278702f9f4120b92121
Summary: There were no unit test for SegmentedChangelogManager so I added one.
Reviewed By: krallin
Differential Revision: D26681459
fbshipit-source-id: 40ceefe7b89043ae6d2c4d31a2adf504245161fb
Summary:
A placeholder for convenience functions.
Right not it has a proxy for the head of the dag.
Reviewed By: krallin
Differential Revision: D26681457
fbshipit-source-id: 6856abbf2685407f96701ea5a508342373503360
Summary:
This is a long standing bug whereby if you made the terminal too thin,
the formatting code would crash.
This fix just prevents underflow in the precision format string (and
alignment).
Reviewed By: fanzeyi
Differential Revision: D26673865
fbshipit-source-id: 945e6f227c19962ac0926e117600aa9d6a990305
Summary:
I accidentally broke this in D26544410 (097e4ad00c) when I updated it to use
schedule_stats_aggregation_preview (in Tokio 0.2 and up, you can't create
an interval stream off the runtime, but in Tokio 0.1 you could).
We only use this method in 2 places, so it probably makes sense to just get rid
of it anyway, which is what this diff does. The alternative is better as it
spawns this unconditionally, so if we get it wrong, it'll fail in tests,
even though our tests don't pass `--fb303-port`, whereas
`start_fb303_and_stats_agg` will only start stats aggregation if its passed.
Reviewed By: ahornby
Differential Revision: D26690223
fbshipit-source-id: 7d151a3c46fa428f00ac32601da161609fb498f7
Summary:
The NFS RPC description is filled with variant over nfsstat3 with only 2 cases:
NFS3_OK, and the default one. I was feeling bad having to write the same code
again and again for the deserialize part of the XdrTrait, and thus came up with
a template solution that enables us to remove all of the boilerplate code.
The only drawback is that this moves the variant one additional layer down, and
that means an extra layer of brace when initializing it. From what I know,
this is caused by the -Wmissing-braces warning, maybe we could remove that
warning?
Reviewed By: kmancini
Differential Revision: D26627163
fbshipit-source-id: ab56dee42273f180b2edf4f529221a15154ac2fb
Summary: This seems to have been reverted by accident in D26618363 (f317302b0f).
Differential Revision: D26689734
fbshipit-source-id: e86451716cab3cc62f517c3f5fca7898d1a25095
Summary:
"value not present in store" is an unhelpful error message once
serialized through Thrift etc. Provide more useful context when a key
is missing.
Reviewed By: fanzeyi
Differential Revision: D26678102
fbshipit-source-id: 514ac2fe580d1dd7c67fc20c89b75e5d8121c329
Summary:
We got a report about windows in vscode being significantly slower than on
macOs/Linux (https://fburl.com/nv9xwc09) to the point that it can't be
explained by "Windows is just slower". As the post suggests we had a suspicion
it might be related to an issue described in T33479254.
After some debugging I can confirm the following:
1) I can reliably (but not 100% of the time) reproduce the issue on my windows
laptop. I just need to modify a file and do "Quick commit" or "Quick amend" from vscode,
and note that command finished quite fast (just a few seconds, I verified it by
running "hg st" and checking that working copy is now clean, and by running "hg
log -r ." and checking that hash has changed), but vscode takes longer to
notice that (i.e. it keeps spinning and showing that amend is still running).
2) By staring at "Process explorer" I noticed that the problem seem to be in
"hg.exe cloud backup" i.e. when "cloud backup" finishes then vscode stops
spinning. So I suspected it to be a problem.
3) As the next step I started to suspend the process in "Process Explorer"
(note that I started it as admin and SYSTEM user, whatever it means). Then I
looked through file handles in Process explorer and was closing them 1 by 1.
Evetually I noticed that closing handles for named pipes like
"\Device\NamedPipe\uv\0000013E91115170-26220" make vscode stop spinning (it was
usually necessary to close it in hg.exe cloud backup process and all child processes as
well).
4) I also looked at handle value, and noticed that it's bigger than 2048 (0x848).
So currently my suspicion is that the problem is that we don't close enought
handles, and this diff bumps it as a temporary workaround and also walk over handles that multiple of 4 only. quark-zju has an
upstream improvement that would make this hack go away
(https://github.com/rust-lang/rust/pull/75551/commits) but it's not landed yet.
So for now let's try to bump the magic number.
Reviewed By: quark-zju
Differential Revision: D26668773
fbshipit-source-id: ed1c203260a52c3e5449b7b06cf4ecbe4dcf6477
Summary:
Previously, when an argument was just a plan filehandle, I just bypassed adding
the structure and just read the handle directly from the stream. kmancini
correctly pointed out that this is a bit confusing, thus let's add the arg
struct everywhere to cut that confusion.
Reviewed By: kmancini
Differential Revision: D26649568
fbshipit-source-id: bc4d6e519450f228d151a3de7acdfc1002c43b38
Summary:
This is merely a forward to the readlink FileInode method. This allows the
.eden directory and its symlinks to be read properly.
Reviewed By: kmancini
Differential Revision: D26619947
fbshipit-source-id: d5f8ef81ce3000c0f0a2f47007affe2c0292ef31
Summary: This adds the various types necessary to implement the READLINK RPC.
Reviewed By: kmancini
Differential Revision: D26610437
fbshipit-source-id: a64644d0499381678170da38886b5dff1c1b9270
Summary:
The LOOKUP RPC is a bit tricky as we need to special case the "." and ".."
names to represent the current and parent directories. Obtaining the parent
directory is inherently racy from the client perspective, and thus EdenFS won't
try to hold the renameLock in this case.
Reviewed By: kmancini
Differential Revision: D26597789
fbshipit-source-id: 9c8dcaf7db84f8c09f7505cab7afed48df79b754
Summary:
See D26671795. The hotfix was ineffective because repo config (and
dynamicconfig) is not loaded. This fix ensures newer versions of
hg do not depend on the hotifx.
Reviewed By: DurhamG
Differential Revision: D26679775
fbshipit-source-id: 5e78051cb7d9ac9ac6b327d895049dca2b025068
Summary:
An OnDemandUpdateDag can now track a bookmark. Every given period it will
query the changeset of the bookmark and incrementally build the dag.
Reviewed By: krallin
Differential Revision: D26656765
fbshipit-source-id: 95057863b5201f9632c654be5544922c7538f974
Summary:
For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present.
Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is.
The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed.
Reviewed By: ikostia
Differential Revision: D26618363
fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
Summary: Can save some allocations and copies by mapping the binary type to bytes::Bytes
Reviewed By: krallin
Differential Revision: D26575835
fbshipit-source-id: 518d2505e89e683d771494b53cb5fae5aea2110b
Summary:
Migrate away from streaming Thrift Python, and instead implement `eden
trace hg` in C++, which will eventually let us support Windows.
This also fixes an unexplained crash in `eden trace hg` on macOS.
Reviewed By: xavierd
Differential Revision: D25515377
fbshipit-source-id: 1916aa7f09df37e9dee3479a858932c724f8aed4
Summary:
We use an EWMA of lag values to ensure that we don't alternate between full speed writes and waiting, but instead give the DB a chance to catch up on replication after writes.
However, in S223394, we saw MyAdmin return extremely high lag values. Because our EWMA is uncapped, and we decay by half each interval, a false spike of 500 seconds takes log2(500 / 5) = 7 seconds to decay back down to the point where we will write our next chunk, assuming that lag is 0 after the false spike (not true, so can be more seconds before lag falls)
Resolve this by capping the EWMA calculation, so that we recover quickly from a false spike (max of 2 seconds with current limit), while still blocking writes while MySQL replication catches up if lag stays high.
Reviewed By: mitrandir77
Differential Revision: D26663528
fbshipit-source-id: 6e5c06f79f08ac6ebbca1b3076816a15ab8353e5
Summary: When building against Python 3.8 the MSVC compiler complains about how this macro splits inside parentheses. Not sure what changed, but looks easy enough to fix.
Reviewed By: quark-zju
Differential Revision: D26434060
fbshipit-source-id: b46059a6e676a52c99f8923b6739cffe578cc6f7
Summary: Supports Windows by adding mkscratch and scmdaemon and updating it to build python3libs.zip instead of python27.zip. Changed the name to include 'libs' on the end since the zip no longer contains Python itself.
Reviewed By: quark-zju
Differential Revision: D26381051
fbshipit-source-id: 0a2cc40df103525fdc581a4102458e82fda1f670
Summary:
In Python 3 the strings are no longer bytes, so we need to convert them before calling Windows native APIs.
I originally attempted to make all C APIs use the W() Windows APIs. This sorta worked and enabled some unicode support for Windows in Python 2, like hg status, but it meant that utf-8 path encodings were being returned and eventually passed to python functions like os.lstat() which weren't expecting utf8 encoding. I gave up and just left the Python 2 C code as is, and made a copy that uses W() APIs for Python 3. This enables unicode support on Windows where it didn't work before (at least in my testing).
Reviewed By: quark-zju
Differential Revision: D26381053
fbshipit-source-id: 69d4e18ba9fb0f3d17bad58fbcc5d0e6e61d4252
Summary:
Windows tests were failing to execute hg clone ssh://user@dummy/... lines because setconfig ui.ssh=C:/foo/bar was being translated to ui.ssh=c;c:\\foo\\bar by mingw, since it detected /foo/bar as a unix-style path. This seems to be caused by this one line reversing the slashes. I'm not sure why it exists, but deleting it makes the tests pass.
My guess is this has been broken for about a year.
Reviewed By: quark-zju
Differential Revision: D26639206
fbshipit-source-id: d89cae1ea3dd055b90ec6ee8f7cdbee2ae08b228
Summary: This diff sets up the ability for us to track requests as they are shuffled around in the `std::vector<> queue`. Since the queue is a vector, and since it is sorted everytime a new element is added or removed, we cannot keep track of elements in the queue with indices or references. Instead, we will store the requests in a shared_ptr so we can maintain a pointer to the request no matter where the request is moved around to.
Reviewed By: kmancini
Differential Revision: D26355907
fbshipit-source-id: d714d689963106a4f495221dbcfcbab758ffc7b2
Summary: The first byte is the flag.
Reviewed By: sfilipco
Differential Revision: D26654640
fbshipit-source-id: 4d66f14c73fe40a154ca7c08f9a9dff3a54ae337
Summary:
Like our other "tokio ecosystem" crates, I'm renaming this one tokio-util-02
(as in tokio-02) since this is the last version it is compatible with. Note that this had a
breaking change as well, so I updated the callsites.
Reviewed By: HarveyHunt
Differential Revision: D26636350
fbshipit-source-id: 30f7da1036c861a97717c8d26648daaae33ecfbd
Summary:
We use a forked version of Gotham in Mononoke. This isn't great, because we
have to maintain this fork. Ideally, we'd upstream our changes, but as is
they're a bit intrusive and not generally useful, which makes this hard.
I've reworked how we do our Gotham changes, and now we only need to make 1 bit
of code public, which might be easier to get upstream. Concretely, Gotham has a
concept of "connected handler" that links a Hyper request and a socket address,
but in our case we want more things. This change lets us instantiate our own
Gotham state, and then add a few more things to it as necessary.
This diff updates our code accordingly to be compatible. This also lets us trim
down on some ceremony we had to do call into Gotham
from Mononoke Server.
Reviewed By: farnz
Differential Revision: D26634653
fbshipit-source-id: 024a48ebc3f323c165ac412ef422755e8cb1c650
Summary:
When in readonly mode we can't write anything, therefore there is no need to monitor for write delay.
Also imported nonzero_ext::nonzero! macro as a small tidy up.
Reviewed By: farnz
Differential Revision: D26662396
fbshipit-source-id: 36cb2c1375a8fb72e3976290907d4c8c34ea0f4e
Summary:
Add case expressions to the templating language that allow selection of a value based on
a single determinator. The expression `case(expr, case1, then1, case2, then2, else)`
will expand to `then1` if `expr` matches `case1`, etc. If it matches none of the cases
then it expands to `else`.
This can be used to simplify long `ifeq` chains in templates.
Reviewed By: quark-zju
Differential Revision: D26631539
fbshipit-source-id: 7543e6f7baa5599c96cac75da17db73e03b918f9
Summary:
I'd like to prepare the migration to Tokio 1.0 and this is one bit of code
that needs some non-trivial changes since in Tokio 1.0, Sleep is no longer
Unpin.
Reviewed By: farnz
Differential Revision: D26610033
fbshipit-source-id: 1db4c1686fcd010e2158bcf4bb25f1e15dd19603
Summary:
Like it says in the title, this updates futures_ext to use tokio_shim, which
makes it compatible with Tokio 0.2 and 1.0.
There is one small difference in behavior here, which is that in Tokio 1.0,
sleep isn't Unpin anymore, so callers will need to call `boxed()` or use Tokio's `pin!` macro if they need
Unpin.
I do want to get as close to what upstream is doing in Tokio 1.0, so I think
it's good to keep that behavior.
Reviewed By: farnz
Differential Revision: D26610036
fbshipit-source-id: ff72275da55558fdf8fe3ad009d25cf84e108a5a
Summary: Make the auth Python bindings match the behavior of `httpconnection.readauthforuri` and omit unset fields in the returned dict.
Reviewed By: quark-zju
Differential Revision: D26439292
fbshipit-source-id: 776d8d9bd59726f5ad8d28973e9c2fdc99a2995c
Summary:
Allow the option of specifying the master tracked bookmark.
(Note: this ignores all push blocking failures!)
Reviewed By: quark-zju
Differential Revision: D26628452
fbshipit-source-id: 10dc4a13d89649355cf6c26b9be91076f2fb6cd3
Summary:
The interesting part is defaults. Particularly around various update periods.
We describe our defaults in metaconfig::types::SegmentedChangelogConfig. An
interesting example is `tailer_update_period`. By default it is
`Some(Duration::from_secs(300))`. Which means that when Segmented Changelog is
enabled we have a pause of 5 minutes before the tailers start the update
process again after an update. We want to prevent the tailer from updating a
specific repository. We do that by using an explicit config which specifies a
duration of 0 seconds. This then becomes an unset (`None`) period in
SegmentedChangelogConfig.
Same idea for all period fields. `update_to_master_bookmark_period` defaults to
`None` currently but we'll update that shortly to `Some(Duration)`.
(Note: this ignores all push blocking failures!)
Reviewed By: quark-zju
Differential Revision: D26628121
fbshipit-source-id: ccfb59b655bf9a7ca3ce2be6c03725eb9b1fe7ed
Summary:
Facebook
It looks like we have quite a few errors when people cloning fbsource on their
laptops (see https://fburl.com/odo8tii0 for more details).
"Couln't resolve host name" is one of the errors we see often, so let's retry
that. ssl errors seem to be retryable as well.
Reviewed By: ikostia
Differential Revision: D26635252
fbshipit-source-id: 2412c4f09aab3d5d45923c4b05fc350556bf9af6
Summary:
It is quite likely that this feature will be more useful if we can increase our
visibility into an entire class of source_hostnames, like mactest or
sandcastlevm, for example.
Let's add support for that.
Reviewed By: krallin
Differential Revision: D26545736
fbshipit-source-id: f84c53969674cee63880fb2cf918bcd87f852dc8
Summary:
The progress suspension is implemented in Rust. Since we're writing to the Rust
"error" and "output" streams, they should handle progress suspension directly.
This also removes the "flakiness" (progress disappears for a while when a line
is printed) when progress is shown in streampager:
lhg debugprogress -s 100 --sleep 100 --with-output --pager=on --config pager.interface=fullscreen
Reviewed By: sfilipco
Differential Revision: D26612478
fbshipit-source-id: b75bfe726c61d89495b198670f777440506fcecf
Summary:
See D23966992 (2a2971a4c7) for context. This is a continuation of D23966995 (ab88771161)
which toggles the deserialization part.
Without this change, the default tuple serialization is inefficient
in CBOR and is a disaster in Python:
In [1]: s,f=api.commithashtolocation('fbsource', repo['master'].node(), [bin('375c7ec7442d2bab635f4ae399be6604c5386eb7')])
In [2]: list(s)
Out[2]:
[{'hgid': (55, 92, 126, 199, 68, 45, 43, 171, 99, 95, 74, 227, 153, 190, 102, 4, 197, 56, 110, 183),
'location': {'descendant': (37, 71, 244, 168, 243, 219, 161, 246, 36, 253, 0, 121, 134, 114, 241, 20, 136, 105, 226, 80),
'distance': 10000}}]
With this change, the Python serialization works as expected:
In [2]: list(s)
Out[2]:
[{'hgid': '7\\~\xc7D-+\xabc_J\xe3\x99\xbef\x04\xc58n\xb7',
'location': {'descendant': '%G\xf4\xa8\xf3\xdb\xa1\xf6$\xfd\x00y\x86r\xf1\x14\x88i\xe2P',
'distance': 10000}}]
Places using HgId serialization directly has opt-in explicit serialization
format: D23966986 (0bb45fcbc4) (zstore), D23966991 (f833f03ba2) (metalog), D24009039 (9c5d20904d) (revisionstore) so
they should not be affected.
EdenAPI protocols used in production should be using `WireHgId` types so they
are not affected.
Reviewed By: sfilipco
Differential Revision: D26562685
fbshipit-source-id: 819b794272ee23a135bbed5b61706944ed0acb9f
Summary:
NFS clients will act differently depending on what kind of error is returned
from an RPC. A NFS3ERR_NOENT in response to a LOOKUP RPC may for instance
populate a negative lookup cache to avoid re-querying the same file again.
Thus, we need to translate the various errors with their corresponding
nfsstat3 values.
Reviewed By: genevievehelsel
Differential Revision: D26597788
fbshipit-source-id: d22a552196a378d6d7e9cd374e63bb8bcf436ce4
Summary:
My edenfs log is filled with the following backtrace:
Traceback (most recent call last):
File "/opt/fb/mercurial/edenscm/mercurial/commands/eden.py", line 421, in get_tree
self._fetch_tree_impl(path, manifest_node)
File "/opt/fb/mercurial/edenscm/mercurial/commands/eden.py", line 671, in _fetch_tree_impl
self.repo._prefetchtrees(path, mfnodes, [], [])
File "/opt/fb/mercurial/edenscm/hgext/treemanifest/__init__.py", line 565, in _prefetchtrees
return self._httpprefetchtrees(*args, **kwargs)
File "/opt/fb/mercurial/edenscm/hgext/treemanifest/__init__.py", line 678, in _httpprefetchtrees
self.edenapi.complete_trees(
TypeError: Expected type that converts to PySequence but received set
And indeed, the code passes a set instead of a plain list.
Reviewed By: kmancini
Differential Revision: D26545708
fbshipit-source-id: 4164154d9cc8991b596198d9324474f646cc194c
