Summary: This diff removes the split between manually managed and autocargo managed Cargo.toml files in `eden/scm/lib`, now all files are autogenerated.
Reviewed By: quark-zju
Differential Revision: D26830884
fbshipit-source-id: 3a5d8409a61347c7650cc7d8192fa426c03733dc
Summary:
On POSIX systems it's a good practice to fsync directories to ensure metadata
about a file is properly written.
Reviewed By: DurhamG
Differential Revision: D26822211
fbshipit-source-id: fca10c702b480f22020ad184eb55c8879999f0ee
Summary:
Memcache is dogslow to initialize, taking >30s on debug build. As a
consequence, this slows down every single test by that amount time, with the
guarantee that no blobs will be found in memcache, ie: a total waste of time.
On release builds, Memcache is significantly faster to initialize, so let's
only disable initializing Memcache for debug builds only.
Reviewed By: fanzeyi
Differential Revision: D26800265
fbshipit-source-id: 8b393c603414de68268fdadb385de177e214a328
Summary:
## Why this diff
we want hostname prefix to support targeting configs at clients in corp ("corp" means laptop, labs, and other machines that are not in "prod" datacenters), like FRL machines, that don't support our existing tier mechanism.
## Changes
* Extract hostname prefix in `dynamicconfig.rs` and add a getter function `hostname_prefix()` for it.
*A hostname prefix only consists of alphabetical letters and dashes, which is followed by one or more digits in the hostname. If no valid match, the prefix is set to the empty string.*
* Use `gen.hostname_prefix()` in the `evaluate()` fn inside `mod.rs` to check the generator's prefix against a list of given prefixes.
* Copy changes from `configerator/source/scm/hg/hgclientconf/hgclient.thrift` to `fbsource/fbcode/configerator/structs/scm/hg/hgclientconf/hgclient.thrift`.
* Rebuild in `eden/scm/`.
Reviewed By: DurhamG
Differential Revision: D26706686
fbshipit-source-id: 725506a1c1f0983e981b0b3f3993c7c14510b1db
Summary: This simplifies the code a bit.
Reviewed By: kulshrax
Differential Revision: D26681779
fbshipit-source-id: 393565790ab711dd09ae6cfa6f9c4b19c930eb93
Summary:
Similar to D26670318, use the EventListeners APIs to implement the progress
callback. Now HandlerExt only has RequestContext related "as_ref" logic.
Reviewed By: kulshrax
Differential Revision: D26681778
fbshipit-source-id: b7f6e07ced43e0ae043e859337c06b69bd5dfc95
Summary: This makes it useful in non-mut callbacks.
Reviewed By: kulshrax
Differential Revision: D26681784
fbshipit-source-id: 97312df8bf3f900a36cbeb27206a2946bb6c702c
Summary: This makes ProgressUpdater Send + Sync so it can be used in the new callback APIs once `mut` gets dropped.
Reviewed By: kulshrax
Differential Revision: D26681781
fbshipit-source-id: 9c622b1d78b4091e3359c28972b6624f0b53734d
Summary:
This removes more mutable fields. Note the new code is more correct because
curl can call the `progress` callback periodically even if no progress is made.
According to https://curl.se/libcurl/c/CURLOPT_PROGRESSFUNCTION.html:
This function gets called by libcurl instead of its internal equivalent with
a frequent interval. While data is being transferred it will be called very
frequently, and during slow periods like when nothing is being transferred it
can slow down to about one call per second.
Reviewed By: kulshrax
Differential Revision: D26681780
fbshipit-source-id: 19aa4bcb4c56623e3f0408b06041b3a894f197e7
Summary:
This makes the `total_progress` field use lock-free interior mutability. The
goal is to eventually drop Rc and RefCell.
Reviewed By: kulshrax
Differential Revision: D26681782
fbshipit-source-id: ec0a6abbb2115c17c674db2255d196aaec847705
Summary: This removes the need for `mut` for this field.
Reviewed By: kulshrax
Differential Revision: D26681783
fbshipit-source-id: 10ed9adfb62081b0e6839abd9534db92d4e056c5
Summary:
The ProgressInner only exposes APIs for total (aggregated) progress. There is
no API to read individual progress. Make it only track the total progress. This
would make it easier to implement interior, lock-free mutability on ProgressInner.
The updater now needs `mut` temporarily, which will be dropped later (D26681784).
A test case is tweaked so progress does not go backwards.
Reviewed By: kulshrax
Differential Revision: D26681777
fbshipit-source-id: 4ad1b9173d5a2c2326e00c030d51f77e9b9458f3
Summary: Test both the HttpClient and Request events.
Reviewed By: kulshrax
Differential Revision: D26670325
fbshipit-source-id: ffbc4268f7de698830411434a769c8b1a4acd863
Summary:
This simplifies the code a bit and makes it look consistent with other event
listeners.
Reviewed By: kulshrax
Differential Revision: D26670318
fbshipit-source-id: f6eda9403bb6eb09a074544e672a45c84f38e2b1
Summary:
Add `RequestContext.event_listeners()` to register callbacks, and trigger the callbacks
when related events happen.
Reviewed By: kulshrax
Differential Revision: D26670323
fbshipit-source-id: 9b92b715444e83957c06b06f1ce696d4de3c0023
Summary:
This simplifies the logic a bit. There is no need for
`HandlerExt::with_payload` that is similar to `Request::body`.
Reviewed By: kulshrax
Differential Revision: D26670326
fbshipit-source-id: 9fe755821062ad6f2a74d6d5ba345620669f0f63
Summary:
This diff rollouts V2 of autocargo in an atomic way so there are quite a few things done here.
Arc lint support:
V1 used to be part of the default fbsource `arc lint` engine, but since V2 calls buck it must live in a separate lint engine. So this diff:
- Adds running `autocargo` as part of `arc lint-rust`
Mergedriver update:
- Mergedriver used in resolving conflicts on commits is now pointing to V2
- It handles files in `public_autocargo/` directories in addition to the ones containig generation preamble
Including regeneration results of running `common/rust/cargo_from_buck/bin/autocargo`. All the differences are accounted for:
- Some sections and attributes are removed as they can be autodiscovered by Cargo (like `lib.path = "src/lib.rs"` or empty [lib] section)
- "readme" attribute is properly defined as relative to Cargo.toml location rather than as hardcoded string
- "unittest = false" on a Buck rule propagates as "test = false; doctest = false" to Cargo
- "rusqlite" is not special-cased anymore, so the "budled" feature will have to be enabled using custom configuration if required by the project (for rust-shed in order to not break windows builds a default feature section was added)
- Files generated from thrift_library rules that do not support "rust" language are removed
- Custom .bzl rules that create rust artifacts (like `rust_python_extension`) are no longer ignored
Others:
- Changed `bin/cargo-autocargo` to be a wrapper for calling V2 via `cargo autocargo`
- Updated following files to use V2:
- `common/rust/tools/reindeer/version-bump`
- `remote_execution/rust/setup.sh`
- Removed few files from V1 that would otherwise interfere with V2 automatic regeneration/linting/testing
Reviewed By: zertosh
Differential Revision: D26728789
fbshipit-source-id: d1454e7ce658a2d3194704f8d77b12d688ec3e64
Summary:
Add a way to define event listeners. Define events on HttpClient and
Request(Info). They will be used by upcoming changes.
The idea is similar to GUI programming. A "control" has a list of "events" that
handlers can be defined on them.
This diff defines lists of events on the "client" and "request" types.
Areas this diff tries to improve:
- Make it easier to add new events (by using macro_rules).
- Make it easier to visually see all possible events.
Reviewed By: kulshrax
Differential Revision: D26670324
fbshipit-source-id: 92f74779f8e546491d0e922db27a4b87f527a5e9
Summary:
This makes it easier to access states defined in `RequestContext` from
`curl::Easy2<H>` types.
Reviewed By: kulshrax
Differential Revision: D26670317
fbshipit-source-id: 27eca9dcc11b14b5d41c8327448f7748ebc62e10
Summary:
Upcoming diffs extend the trait with new methods unrelated to configuration.
Rename to clarify.
Reviewed By: kulshrax
Differential Revision: D26670314
fbshipit-source-id: 7d33ebe22b26f1a286ae40c78f51f31a1a64957e
Summary:
Similar to the Streaming curl handler, add a RequestContext field to the Buffered
handler so the curl callbacks on the Handler can provide the Request
information like urls.
Reviewed By: kulshrax
Differential Revision: D26670321
fbshipit-source-id: de7abecf162c4aaed03d927c35516b6f8ac523ce
Summary:
Make `RequestContext` available in the streaming request. The `clone` will
be removed by a later change.
`dead_code` is temporarily allowed so the following won't be an error.
error: field is never read: `request_info`
--> src/handler/streaming.rs:24:5
|
24 | request_info: RequestContext,
| ^^^^^^^^^^^^^^^^^^^^^^^^^
|
Reviewed By: kulshrax
Differential Revision: D26670319
fbshipit-source-id: 53a1deeece5a2059e7caa9d28ef00e083a27b722
Summary:
Add an "auto incremental" Id to uniquely identify requests. This allows external
logic to have a map from the Id to extra metadata not owned by this crate. Without
the Id, there is no way to tell if `RequestContext`s with a same url and method are
actually a same request or not.
Reviewed By: kulshrax
Differential Revision: D26670327
fbshipit-source-id: 60fa760432b23ab5334f22806e01304f9c160182
Summary:
The `RequestContext` is a subset of `Request` that are:
- Independent from curl types.
- Carry useful states, and make them available on Handler callbacks.
- For now, the "useful states" include url and method. They can be extended
later.
Reviewed By: kulshrax
Differential Revision: D26670320
fbshipit-source-id: 3d71d5fee8927dd57a52f51b212397710379e7fa
Summary: This treats update_meta collection same way as update_content
Reviewed By: DurhamG
Differential Revision: D26694077
fbshipit-source-id: f2a7afb988eb99afa6e8f62ae6cda57a108a987d
Summary: This diff adds unit test for sparse checkout plan
Reviewed By: DurhamG
Differential Revision: D26673701
fbshipit-source-id: 8f810df317c3447f8445765e3a510559d4f1b378
Summary:
This method updates checkout plan to account for sparse profile change
The logic is somewhat similar to sparse.py, except it is smaller because it does not account for merges
Reviewed By: DurhamG
Differential Revision: D26670413
fbshipit-source-id: 54294ad7f1c61e39400450ff139f40ac8fffdd62
Summary:
This matcher combinator is useful when checking out across sparse profile change, since it allows to list files that has been affected by the sparse profile chane
Python has similar combinator xormatcher
Reviewed By: DurhamG
Differential Revision: D26670412
fbshipit-source-id: 0f84835aad16177e79e87d15b41b2caef4374605
Summary:
There are some reports about truncated latest files on NFS hosts (where the
hgcache portion might be outside NFS).
Fsync files in hgcache too. With indexedlog the total number of files is
bounded so listing them is considered bounded time.
Reviewed By: kulshrax
Differential Revision: D26741889
fbshipit-source-id: a789396cc5802110eb55fa6b52b5124cd9a3dbf3
Summary: It seems there is no "tty" check. So let's disable colors for tests explicitly.
Reviewed By: DurhamG
Differential Revision: D26708620
fbshipit-source-id: 802537af0d3f580c39a746b6aa6a617c9fb4c9c1
Summary: A few new config options were added to `EdenApiBuilder` without corresponding setting methods, meaning that they could only be set via an hgrc config file. This diff adds the methods so that Rust code can manually configure these settings.
Differential Revision: D26709471
fbshipit-source-id: 6af645961ed50526ccc04990b82ee201211b6ad2
Summary: Add `Content-Length` to `ResponseMeta` struct. Will be used for logging purposes.
Differential Revision: D26709386
fbshipit-source-id: 4346627e3883d00350738cca1e71004f8d407c0d
Summary:
Currently running native checkout shows much higher sys time then regular checkout
One theory that this is due to high scheduler overhead due to large number small tasks spawned with spawn_blocking
This diff batches 100 operations before spawning blocking proc
Reviewed By: quark-zju
Differential Revision: D26620395
fbshipit-source-id: f5be84c3ee14ffc55e4332e714a48f55701d5f8f
Summary:
Recently there are reports about "remote/master" pointing to a non-existed
commit, missing tree data, and general hard-reboot issues. Attempt to reduce
the chance of losing data by adding the critical paths to the fsync list.
Reviewed By: kulshrax
Differential Revision: D26708225
fbshipit-source-id: ae52534fdbf8d9d35498d30c24bdce7f5d9a96a3
Summary:
We got a report about windows in vscode being significantly slower than on
macOs/Linux (https://fburl.com/nv9xwc09) to the point that it can't be
explained by "Windows is just slower". As the post suggests we had a suspicion
it might be related to an issue described in T33479254.
After some debugging I can confirm the following:
1) I can reliably (but not 100% of the time) reproduce the issue on my windows
laptop. I just need to modify a file and do "Quick commit" or "Quick amend" from vscode,
and note that command finished quite fast (just a few seconds, I verified it by
running "hg st" and checking that working copy is now clean, and by running "hg
log -r ." and checking that hash has changed), but vscode takes longer to
notice that (i.e. it keeps spinning and showing that amend is still running).
2) By staring at "Process explorer" I noticed that the problem seem to be in
"hg.exe cloud backup" i.e. when "cloud backup" finishes then vscode stops
spinning. So I suspected it to be a problem.
3) As the next step I started to suspend the process in "Process Explorer"
(note that I started it as admin and SYSTEM user, whatever it means). Then I
looked through file handles in Process explorer and was closing them 1 by 1.
Evetually I noticed that closing handles for named pipes like
"\Device\NamedPipe\uv\0000013E91115170-26220" make vscode stop spinning (it was
usually necessary to close it in hg.exe cloud backup process and all child processes as
well).
4) I also looked at handle value, and noticed that it's bigger than 2048 (0x848).
So currently my suspicion is that the problem is that we don't close enought
handles, and this diff bumps it as a temporary workaround and also walk over handles that multiple of 4 only. quark-zju has an
upstream improvement that would make this hack go away
(https://github.com/rust-lang/rust/pull/75551/commits) but it's not landed yet.
So for now let's try to bump the magic number.
Reviewed By: quark-zju
Differential Revision: D26668773
fbshipit-source-id: ed1c203260a52c3e5449b7b06cf4ecbe4dcf6477
Summary: The first byte is the flag.
Reviewed By: sfilipco
Differential Revision: D26654640
fbshipit-source-id: 4d66f14c73fe40a154ca7c08f9a9dff3a54ae337
Summary:
Facebook
It looks like we have quite a few errors when people cloning fbsource on their
laptops (see https://fburl.com/odo8tii0 for more details).
"Couln't resolve host name" is one of the errors we see often, so let's retry
that. ssl errors seem to be retryable as well.
Reviewed By: ikostia
Differential Revision: D26635252
fbshipit-source-id: 2412c4f09aab3d5d45923c4b05fc350556bf9af6
Summary:
See D23966992 (2a2971a4c7) for context. This is a continuation of D23966995 (ab88771161)
which toggles the deserialization part.
Without this change, the default tuple serialization is inefficient
in CBOR and is a disaster in Python:
In [1]: s,f=api.commithashtolocation('fbsource', repo['master'].node(), [bin('375c7ec7442d2bab635f4ae399be6604c5386eb7')])
In [2]: list(s)
Out[2]:
[{'hgid': (55, 92, 126, 199, 68, 45, 43, 171, 99, 95, 74, 227, 153, 190, 102, 4, 197, 56, 110, 183),
'location': {'descendant': (37, 71, 244, 168, 243, 219, 161, 246, 36, 253, 0, 121, 134, 114, 241, 20, 136, 105, 226, 80),
'distance': 10000}}]
With this change, the Python serialization works as expected:
In [2]: list(s)
Out[2]:
[{'hgid': '7\\~\xc7D-+\xabc_J\xe3\x99\xbef\x04\xc58n\xb7',
'location': {'descendant': '%G\xf4\xa8\xf3\xdb\xa1\xf6$\xfd\x00y\x86r\xf1\x14\x88i\xe2P',
'distance': 10000}}]
Places using HgId serialization directly has opt-in explicit serialization
format: D23966986 (0bb45fcbc4) (zstore), D23966991 (f833f03ba2) (metalog), D24009039 (9c5d20904d) (revisionstore) so
they should not be affected.
EdenAPI protocols used in production should be using `WireHgId` types so they
are not affected.
Reviewed By: sfilipco
Differential Revision: D26562685
fbshipit-source-id: 819b794272ee23a135bbed5b61706944ed0acb9f
Summary:
The Rust IO handles progress and streampager stuff. Switch to it so we don't
need to changing the `fout`, `ferr` when handling streampager in Python.
The chgserver logic is updated to just set raw fd 0, 1, 2 to update stdio,
since `fileno` is no longer exposed from Rust.
Manually tested the following commands, both without chg and with chg:
- lhg log -r . (no pager)
- lhg log (with streampager)
- lhg log --config pager.pager=less (with less pager)
- lhg commit (spawns pager)
- lhg debugprogress -s 100 --sleep 100 --with-output --pager=off (progress in stderr)
- lhg debugprogress -s 100 --sleep 100 --with-output --pager=on --config pager.interface=fullscreen (progress in streampager)
- lhg debugprogress -s 100 --sleep 100 --with-output --pager=on --config pager.pager='LESS= less' (progress is disabled with external pager)
Reviewed By: sfilipco
Differential Revision: D26612487
fbshipit-source-id: 8b4e36b614a0c080b93e41474f9a8fc33f890083
Summary:
Add a new API to test if a stream is a tty. This is needed to replace the
Python `fin`, `fout` etc. to Rust objects, because the Python land requires the
`istty` API. It is also useful for properly implement color detection in the
pure Rust land.
Reviewed By: sfilipco
Differential Revision: D26612480
fbshipit-source-id: 5cf79447b1d74e0031a954788db342afd48dc288
Summary:
Move IO to a separate crate so it is easier to be used in other libraries.
Practically, I'm going to add `is_tty()` API, and make some types from
`cpython-ext` implement it. If `cpython-ext` depends on `clidispatch`,
that would be too heavyweight.
Reviewed By: sfilipco
Differential Revision: D26612488
fbshipit-source-id: 00b18dadce3157a357d189aaa7d985efa6b4c80a
Summary:
I'm going to expose more APIs from the Rust IO such as setting progress
content, etc. Rename the module to clarify.
Reviewed By: sfilipco
Differential Revision: D26612490
fbshipit-source-id: 136ea27a733b09557d02077e68ce51184125ade1
