Summary: Running these with tsan appears to run properly, let's try to re-enable them.
Reviewed By: genevievehelsel
Differential Revision: D27723525
fbshipit-source-id: 42e61d26cf478cbe808698a6a0615015832180fa
Summary:
We have `experimental.findcommonheadsnew` set to True in all tests, and
Rust commit backends force the `findcommonheadsnew` paths, which is
pretty much everywhere except hgsql repos. Remove `_findcommonheadsold`.
The fast discovery is also unnecessary. Remove them too.
Reviewed By: DurhamG
Differential Revision: D27630496
fbshipit-source-id: ab1948f03a8c84e75e3b5c9ff4769e17533447d2
Summary: Many users have asked what scratch directory is when they look to free some disk space. Placing a README.txt under the scratch root could be helpful to explain what is in there.
Reviewed By: fanzeyi
Differential Revision: D27710277
fbshipit-source-id: e3ccd92fa1920ac4c791026b8d98aa05a1c8b268
Summary: This diff uses filescmstore for native checkout when nativecheckout.usescmstore config is set
Reviewed By: DurhamG
Differential Revision: D27658844
fbshipit-source-id: ec3442d677ccb25e8b08cc194e4c8c18c0e01fa1
Summary:
This diff introduces CheckoutPlan::apply_read_store to apply checkout plan using ReadStore as a data source
This requires some minor changes in apply_stream flow as ReadStore does not guarantee ordering of returned files
Reviewed By: DurhamG
Differential Revision: D27658346
fbshipit-source-id: 5a289554d8dd7b6bb4b5a996659cd0661779ad5f
Summary: The latter is more lightweight.
Reviewed By: DurhamG
Differential Revision: D27641665
fbshipit-source-id: d46f62f9067eb9cb4c8517a62efa6f663d4b6732
Summary: The latter is more lightweight.
Reviewed By: DurhamG
Differential Revision: D27641669
fbshipit-source-id: d907407f5a6e868862fe37f1f67fbe99ee378156
Summary: The latter is more lightweight.
Reviewed By: DurhamG
Differential Revision: D27641667
fbshipit-source-id: adce5a39fcb5d8e8d5d989fed46991e20ab3710d
Summary: Provides a way to read config with lighter dependencies.
Reviewed By: DurhamG
Differential Revision: D27641668
fbshipit-source-id: fc99a78f5f51e63f61d1b049af74f61f5d1916a3
Summary:
The `configparser` is now too heavyweight. Some other crates (ex. io, auth,
revisionstore) just want to extract config values without complicated parsing /
remote hotfix logic.
Add a configmodel crate to satisfy the need.
Reviewed By: DurhamG
Differential Revision: D27641666
fbshipit-source-id: 26bd0b606ae3d286b3ec218927aef726d6802c63
Summary:
The CMake documentation states:
>By default, the value index of value-parameterized tests is replaced by the
>actual value in the CTest test name. If this behavior is undesirable (e.g.
>because the value strings are unwieldy), this option will suppress this
>behavior.
Which appears to be a decent default, but not when the parameter is a pointer,
in which case the test name will contain some hex values.
Reviewed By: genevievehelsel
Differential Revision: D27713222
fbshipit-source-id: 0f15b24d04817384ff975ad7b07e16b744e1eb2e
Summary: Add a new `http.verbose` config option that turns on verbose output for libcurl (similar to the output printed by `curl -v`). This can be very useful for debugging HTTP issues.
Reviewed By: DurhamG
Differential Revision: D27693304
fbshipit-source-id: 2ad7a08889f40ffbcd2f14ac9c21d70433629da4
Summary: Make this method match the behavior of `remotefileslog.memcachestore` and cache the `edenapistore` instead of constructing a new one each time. Right now this doesn't matter too much because we currently only call this once when setting up the Rust `revisionstore`, but it would be good to avoid creating multiple instances if we do start using this elsewhere.
Reviewed By: DurhamG
Differential Revision: D27684210
fbshipit-source-id: 7987f603c79758902b4740dd8b46d26a25baec93
Summary:
This diff causes memcache to be disabled when `remotefilelog.cachekey` is set to the empty string, thereby allowing memcache to be disabled on the command line with `--config remotefilelog.cachekey=''`. This is useful when testing data-layer changes.
Previously, the only way to do this was to add `%unset cachekey` to the `remotefilelog` section of your `hgrc`, which was a bit tedious compared to just using a `--config` flag.
Reviewed By: DurhamG
Differential Revision: D27683782
fbshipit-source-id: 3e0434e98a32db916a07935e8b26f70317f50286
Summary: The error is not going to have a message in many cases leading to crashes.
Reviewed By: quark-zju
Differential Revision: D27637119
fbshipit-source-id: 135b133371916dddf0c47a84f00957a8b8fdfe92
Summary:
It looks like nfs_test is tripping the nightly pyre infer job causing it to not
publish diff that increase our typing coverage. Let's manually type it to
unblock the nightly job.
Reviewed By: genevievehelsel
Differential Revision: D27682093
fbshipit-source-id: a32df9f5b8eeaef2006de7d64f5adadb763402e8
Summary:
We hammer MySQL during GC - slow down so that bad connections to
servers that are no longer current are dropped from the pool
https://fb.workplace.com/groups/scm.mononoke/permalink/1407064449656126/?comment_id=1408378062858098 justifies setting the MySQL max ages to 1 second - it works around a MyRouter issue where it *should* reconnect us to a different host, but doesn't.
Reviewed By: krallin
Differential Revision: D27500583
fbshipit-source-id: e900925e1f0d65828613fe3e3d7f4128dc7cde82
Summary: SQLBlob doesn't benefit from sharing a pool with other MySQL users, but does benefit from more aggressive connection timeouts. Give it its own pool, which we can tweak later.
Reviewed By: krallin
Differential Revision: D27651133
fbshipit-source-id: 8f5216ec0506b217f9365babfe1ebac00f68a9a9
Summary:
Like it says in the title. This is a place where we use timeseries so we might
as well use that shared crate.
Reviewed By: mzr
Differential Revision: D27678389
fbshipit-source-id: 9b5d4980a1ddb5ce2a01c8ef417c78b1c3da80b7
Summary:
I'd like to be able to track time series for access within Mononoke. The
underlying use case here is that I want to be able to track the max count of
connections in our SQL connection pools over time (and possibly other things in
the future).
Now, the obvious question is: why am I rolling my own? Well, as it turns out,
there isn't really an implementation of this that I can reuse:
- You might expect to be able to track the max of a value via fb303, but you
can't:
https://www.internalfb.com/intern/diffusion/FBS/browse/master/fbcode/fb303/ExportType.h?commit=0405521ec858e012c0692063209f3e13a2671043&lines=26-29
- You might go look in Folly, but you'll find that the time series there only
supports tracking Sum & Average, but I want my timeseries to track Max (and
in fact I'd like it to be sufficiently flexible to track anything I want):
https://www.internalfb.com/intern/diffusion/FBS/browse/master/fbcode/folly/stats/BucketedTimeSeries.h
It's not the first time I've ran into a need for something like this. I needed
it in RendezVous to track connections over the last 2 N millisecond intervals,
and we needed it in metagit for host draining as well (note that the
implementation here is somewhat inspired by the implementation there).
Reviewed By: mzr
Differential Revision: D27678388
fbshipit-source-id: ba6d244b8bb848d4e1a12f9c6f54e3aa729f6c9c
Summary:
This is breaking with a warning because there's a method called `intersperse`
that might be introduced in the std lib:
```
stderr: error: a method with this name may be added to the standard library in the future
--> eden/mononoke/hgproto/src/sshproto/response.rs:48:53
|
48 | let separated_results = escaped_results.intersperse(separator);
| ^^^^^^^^^^^
|
note: the lint level is defined here
--> eden/mononoke/hgproto/src/lib.rs:14:9
```
This should fix it.
Reviewed By: ikostia
Differential Revision: D27705212
fbshipit-source-id: 5f2f641ea6561c838288c8b158c6d9e134ec0724
Summary:
`const_cstr::ConstCStr` is represented internally as a fat pointer with fixed size: `&'static str`. See https://docs.rs/const-cstr/0.3.0/const_cstr/struct.ConstCStr.html. Notably this is **different** from the representation of `std::ffi::CStr`, which is a dynamically sized type and normally passed around behind a reference, as `&CStr`. Using `&ConstCStr` in signatures, which is effectively like `&'a &'static CStr`, is confusing due to the discrepancy between the two relatedly named types. Additionally having two different lifetimes involved -- the static lifetime of the underlying bytes, and the short lifetime of the fat pointer -- is unnecessarily confusing when async code and a language boundary are involved.
The utf8-cstr crate uses what seems like a better representation to me than the const-cstr crate. See https://docs.rs/utf8-cstr/0.1.6/utf8_cstr/struct.Utf8CStr.html. `Utf8CStr` is the dynamically sized type, just like `CStr`. Then `&'static Utf8CStr` is how it would commonly be passed around, just like `&CStr`.
Reviewed By: krallin
Differential Revision: D27698169
fbshipit-source-id: ffe172c2c2fc77aeab6b0a0a8aed3e3c196098cc
Summary: Migrate the codebase away from the ad-hoc `folly::uint64ToBufferUnsafe` and to `folly::to_ascii_decimal` which is intended for these cases.
Reviewed By: WillerZ
Differential Revision: D27281875
fbshipit-source-id: 0c98749e4aed9c873853eed2221cf54a89279ff4
Summary:
The hashes that are passed in as parameters to the hash-to-location function
may not be hashes that actually exist. This change updates the code so that
we don't return an error when an unknown hash is passed in. The unknown
hash will be skipped in the list of results.
Reviewed By: quark-zju
Differential Revision: D27526758
fbshipit-source-id: 8bf9b7a134a6a8a4f78fa0df276f847d922472f5
Summary:
We want to handle the case where the client has multiple heads for master. For
example when master is moved backwards (or when it get moved on the client by
force). Updating the client code to thread the list of master commits to the
EdenApi client.
Reviewed By: quark-zju
Differential Revision: D27523868
fbshipit-source-id: db4148a3f1d0e8b0b162e0ecc934e07f041c5511
Summary:
We want to handle the case where the client has multiple heads for master. For
example when master is moved backwards (or when it get moved on the client by
force). Updating the request object for HashToLocation to send over all the
master heads.
When the server builds non-master commits, we will want to send over non-master
heads too. We may consider having one big list of heads but I think that we
would still want to distinguish the non-master commit case in order to optimize
that use-case.
Reviewed By: quark-zju
Differential Revision: D27521778
fbshipit-source-id: cc83119b47ee90f902c186528186ad57bf023804
Summary:
This scenario appears when master moves backwards. Since the master group in
segmented changelog is append-only. Non-fast-forward master move will cause
multiple heads in the master group.
Since Segmented Changelog was updated to handle multiple master heads, we can
propagate the full list that we get from the client.
This diff makes the assumption that Mononoke will know to convert all client "master head"
hashes from HgChangesetId (Sha1) form to ChangesetId (Blake2). If any of the master
heads cannot be converted then it means the server might not be reliably answer the
client's question (in "ancestors(master_heads)", translate "this hash" to a path, or tell me
confidently that the "hash" is outside "ancestors(master_heads)"). That's an error case.
Reviewed By: quark-zju
Differential Revision: D27521779
fbshipit-source-id: 219e08a66aac17ac06d2cf02676a43c7f37e8e26
Summary:
This scenario appears when master moves backwards.
Since the IdDag can handle multiple master heads, the server can piggy-back on that
functionality and support multiple master heads when translating location to hash.
Reviewed By: quark-zju
Differential Revision: D27521780
fbshipit-source-id: c27541890d4fda13648857f010c11a25bf96ef67
Summary:
Fixes needed:
- buck2/linter commented out - unstable API change
- hermetic/reverie - ExitStatusExt sealed (fixed in D27637862)
- Iterator::fold_first stabilized as reduce
- panic!() requires literal format string (fixed in D27672891)
I didn't change anything under scripts/ - a lot of it looked like it was already broken.
Reviewed By: dtolnay
Differential Revision: D27654989
fbshipit-source-id: d1eac6e51686dfb74c87cec8718f075066df4b8f
Summary:
`panic!()`, and things which use `panic!()` like `assert!()`, take a literal format
string, and whatever parameters to format. There's no need to use `format!()`
with it, and it is incorrect to pass a non-literal string.
Mostly it's harmless, but there are some genuinely confusing asserts which
trigger this warning.
Reviewed By: dtolnay
Differential Revision: D27672891
fbshipit-source-id: 73929cc77c4b3d354bd315d8d4b91ed093d2946b
Summary:
Modify the `Debug` implementation for `minibytes::Bytes` to use `std::ascii::escape_default` to debug print a `Bytes` as an ascii-escaped bytestring.
For comparison, the `bytes` crate `Bytes` type provides the same functionality, though it doesn't use the standard library `escape_default` function: https://docs.rs/bytes/1.0.1/src/bytes/fmt/debug.rs.html#39-43
This change greatly improves the output of the `debugscmstore` command. If we don't want to make this the default behavior, we can provide a formatting wrapper type or I can specialize the output in `debugscmstore`, but I can't see any real downsides, especially given the `bytes` crate does the same thing, and we have a similar specialization for `HgId` (hex format in that case).
Reviewed By: quark-zju
Differential Revision: D27642721
fbshipit-source-id: 8faba421fa5082a2098b13ef7d286e05eccb6400
Summary: Add the `with_key` function to `Entry`, which replaces it's key with a provided key. Currently, scmstore returns incorrect results when multiple entries exist with different paths but the same HgId (as scmstore directly returns the path found on disk locally). This isn't a problem in the legacy API, which returns a bare `Vec<u8>` content, which is implicitly associated with the requesting key because it is the result of a single `get` call, or is irrelevant because the `prefetch` method doesn't directly return the results.
Reviewed By: andll
Differential Revision: D27664025
fbshipit-source-id: 014d44ca9a1dc2721685622fd2b077ed3483838f
Summary:
We are moving our skelton for eden top a bit further, by creating the view
structure for eden top. This creates the widget for the banner of eden top,
the help page and the main page as well as a couple sections of the main
page.
The main page is displayed on default and they are toggled with `h` for help
page and `esc` to return to the mage page. Visable and hidden widgets are not
implemented in termwiz yet so we have to do a bit of hacking to hide and
display widgets for our selves.
Each of the sections is stubed with place holder text for testing.
Reviewed By: xavierd
Differential Revision: D26892620
fbshipit-source-id: a7bb4d0e11f3a8968ef071e7f585d07a9c286880
Summary:
D27659634 (8e8aaa61d6) removed these files, so let's drop their exclusions from
test-check-code.t
Reviewed By: sfilipco
Differential Revision: D27682136
fbshipit-source-id: f8e10fac37ea90fb2782b960faf4536f1ff9133b
Summary:
fctx is not guaranteed to have the _path and _filenode attributes. Those are
specific to implementations, e.g. `absentfilectx` does not have them.
`basefilectx` instead defines the `path()` and `filenode()` for general fctx
use.
Reviewed By: quark-zju
Differential Revision: D27667176
fbshipit-source-id: 1d7889d264b597665ef05f84a752323f078cb455
Summary:
Create a fork of the Mercurial code that we can use to build server
rpms. The hg servers will continue to exist for a few more months while we move
the darkstorm and ediscovery use cases off them. In the mean time, we want to
start making breaking changes to the client, so let's create a stable copy of
the hg code to produce rpms for the hg servers.
The fork is based off c7770c78d, the latest hg release.
This copies the files as is, then adds some minor tweaks to get it to build:
- Disables some lint checks that appear to be bypassed by path
- sed replace eden/scm with eden/hg-server
- Removed a dependency on scm/telemetry from the edenfs-client tests since
scm/telemetry pulls in the original eden/scm/lib/configparser which conflicts
with the hg-server conflict parser.
allow-large-files
Reviewed By: quark-zju
Differential Revision: D27632557
fbshipit-source-id: b2f442f4ec000ea08e4d62de068750832198e1f4
Summary:
as titled, an ugly hack to support LABEL attribute for 1.7.0 which didn't exist.
getting rid of it now since oss support upgraded to 1.7.1
Differential Revision: D27536730
fbshipit-source-id: 58653e71cbf20a1cda0b7414f66b5f0014d89b84
Summary:
Sometimes we can hit an Idle timeout error while talking to MySQL, because we open a connection and go idle for a long time. Then when we finally send a query, server returns an error: the connection is expired. This is the issue we found and fixed in D27503062 (a856799489) that blocked MySQL client release.
## Future starvation
Imagine you have a stream in which you're connecting to a server, fetching and preparing some values:
```
let v = vec![1u32, 2, 3, 4, 5];
let mut s = stream::iter(v)
.map(|key| async move {
let conn = connect(..).await?;
conn.fetch(..).await
})
.buffered(2)
.map(|item| async move { prepare(..) })
.buffered(2);
```
Now you want to asynchronously process those prepared values one by one:
```
while let Some(result) = s.next().await {
let value = result?;
process(value).await?;
}
```
This async `process(..)` call can be talking to some service to take these values or something else that doesn't require much of a CPU time. Although the operation can be long.
**Now what happens when we do s.next().await?**
Because the stream is `buffered(2)` we wait for the first 2 futures. When the first item is ready, it returns the result and polls next stream item with a key - 3. The third future only makes a `connect(..)` call and gets switched.
When we've got a next value from the stream, we're waiting on the `process(value)` call and not polling the underlying stream till the processing is done.
**As I mentioned earlier, it is not expensive...**
But what if it takes > 10s to complete anyway?
The third future from the stream, that was polled earlier, **will wait for all these > 10s till it is polled again**.
More details [in this post](https://fb.workplace.com/groups/learningrust/permalink/2890621307875402/).
## Solution
In this case spawning a future with connection and query steps is a way to fix the issue.
This diff spawns queries in `shed::sql::sql_common::mysql_wrapper` - this covers all the places in Mononoke where we talk to MySQL. Also I removed the spawn from hg sync code, because it is not needed anymore and to illustrate that this approach works.
Reviewed By: StanislavGlebik
Differential Revision: D27639629
fbshipit-source-id: edaa2ce8f5948bf44e1899a19b443935920e33ef
Summary:
`getdeps` builds sometimes fail - try a higher compression level to
see if it's a different default internally and in open source
Reviewed By: ahornby
Differential Revision: D27659420
fbshipit-source-id: 702341e6b288ab79584bfa8de5b1ccd5ed6bc57a
