Summary:
The CMake documentation states:
>By default, the value index of value-parameterized tests is replaced by the
>actual value in the CTest test name. If this behavior is undesirable (e.g.
>because the value strings are unwieldy), this option will suppress this
>behavior.
Which appears to be a decent default, but not when the parameter is a pointer,
in which case the test name will contain some hex values.
Reviewed By: genevievehelsel
Differential Revision: D27713222
fbshipit-source-id: 0f15b24d04817384ff975ad7b07e16b744e1eb2e
Summary: Add a new `http.verbose` config option that turns on verbose output for libcurl (similar to the output printed by `curl -v`). This can be very useful for debugging HTTP issues.
Reviewed By: DurhamG
Differential Revision: D27693304
fbshipit-source-id: 2ad7a08889f40ffbcd2f14ac9c21d70433629da4
Summary: Make this method match the behavior of `remotefileslog.memcachestore` and cache the `edenapistore` instead of constructing a new one each time. Right now this doesn't matter too much because we currently only call this once when setting up the Rust `revisionstore`, but it would be good to avoid creating multiple instances if we do start using this elsewhere.
Reviewed By: DurhamG
Differential Revision: D27684210
fbshipit-source-id: 7987f603c79758902b4740dd8b46d26a25baec93
Summary:
This diff causes memcache to be disabled when `remotefilelog.cachekey` is set to the empty string, thereby allowing memcache to be disabled on the command line with `--config remotefilelog.cachekey=''`. This is useful when testing data-layer changes.
Previously, the only way to do this was to add `%unset cachekey` to the `remotefilelog` section of your `hgrc`, which was a bit tedious compared to just using a `--config` flag.
Reviewed By: DurhamG
Differential Revision: D27683782
fbshipit-source-id: 3e0434e98a32db916a07935e8b26f70317f50286
Summary: The error is not going to have a message in many cases leading to crashes.
Reviewed By: quark-zju
Differential Revision: D27637119
fbshipit-source-id: 135b133371916dddf0c47a84f00957a8b8fdfe92
Summary:
It looks like nfs_test is tripping the nightly pyre infer job causing it to not
publish diff that increase our typing coverage. Let's manually type it to
unblock the nightly job.
Reviewed By: genevievehelsel
Differential Revision: D27682093
fbshipit-source-id: a32df9f5b8eeaef2006de7d64f5adadb763402e8
Summary:
We hammer MySQL during GC - slow down so that bad connections to
servers that are no longer current are dropped from the pool
https://fb.workplace.com/groups/scm.mononoke/permalink/1407064449656126/?comment_id=1408378062858098 justifies setting the MySQL max ages to 1 second - it works around a MyRouter issue where it *should* reconnect us to a different host, but doesn't.
Reviewed By: krallin
Differential Revision: D27500583
fbshipit-source-id: e900925e1f0d65828613fe3e3d7f4128dc7cde82
Summary: SQLBlob doesn't benefit from sharing a pool with other MySQL users, but does benefit from more aggressive connection timeouts. Give it its own pool, which we can tweak later.
Reviewed By: krallin
Differential Revision: D27651133
fbshipit-source-id: 8f5216ec0506b217f9365babfe1ebac00f68a9a9
Summary:
Like it says in the title. This is a place where we use timeseries so we might
as well use that shared crate.
Reviewed By: mzr
Differential Revision: D27678389
fbshipit-source-id: 9b5d4980a1ddb5ce2a01c8ef417c78b1c3da80b7
Summary:
I'd like to be able to track time series for access within Mononoke. The
underlying use case here is that I want to be able to track the max count of
connections in our SQL connection pools over time (and possibly other things in
the future).
Now, the obvious question is: why am I rolling my own? Well, as it turns out,
there isn't really an implementation of this that I can reuse:
- You might expect to be able to track the max of a value via fb303, but you
can't:
https://www.internalfb.com/intern/diffusion/FBS/browse/master/fbcode/fb303/ExportType.h?commit=0405521ec858e012c0692063209f3e13a2671043&lines=26-29
- You might go look in Folly, but you'll find that the time series there only
supports tracking Sum & Average, but I want my timeseries to track Max (and
in fact I'd like it to be sufficiently flexible to track anything I want):
https://www.internalfb.com/intern/diffusion/FBS/browse/master/fbcode/folly/stats/BucketedTimeSeries.h
It's not the first time I've ran into a need for something like this. I needed
it in RendezVous to track connections over the last 2 N millisecond intervals,
and we needed it in metagit for host draining as well (note that the
implementation here is somewhat inspired by the implementation there).
Reviewed By: mzr
Differential Revision: D27678388
fbshipit-source-id: ba6d244b8bb848d4e1a12f9c6f54e3aa729f6c9c
Summary:
This is breaking with a warning because there's a method called `intersperse`
that might be introduced in the std lib:
```
stderr: error: a method with this name may be added to the standard library in the future
--> eden/mononoke/hgproto/src/sshproto/response.rs:48:53
|
48 | let separated_results = escaped_results.intersperse(separator);
| ^^^^^^^^^^^
|
note: the lint level is defined here
--> eden/mononoke/hgproto/src/lib.rs:14:9
```
This should fix it.
Reviewed By: ikostia
Differential Revision: D27705212
fbshipit-source-id: 5f2f641ea6561c838288c8b158c6d9e134ec0724
Summary:
`const_cstr::ConstCStr` is represented internally as a fat pointer with fixed size: `&'static str`. See https://docs.rs/const-cstr/0.3.0/const_cstr/struct.ConstCStr.html. Notably this is **different** from the representation of `std::ffi::CStr`, which is a dynamically sized type and normally passed around behind a reference, as `&CStr`. Using `&ConstCStr` in signatures, which is effectively like `&'a &'static CStr`, is confusing due to the discrepancy between the two relatedly named types. Additionally having two different lifetimes involved -- the static lifetime of the underlying bytes, and the short lifetime of the fat pointer -- is unnecessarily confusing when async code and a language boundary are involved.
The utf8-cstr crate uses what seems like a better representation to me than the const-cstr crate. See https://docs.rs/utf8-cstr/0.1.6/utf8_cstr/struct.Utf8CStr.html. `Utf8CStr` is the dynamically sized type, just like `CStr`. Then `&'static Utf8CStr` is how it would commonly be passed around, just like `&CStr`.
Reviewed By: krallin
Differential Revision: D27698169
fbshipit-source-id: ffe172c2c2fc77aeab6b0a0a8aed3e3c196098cc
Summary: Migrate the codebase away from the ad-hoc `folly::uint64ToBufferUnsafe` and to `folly::to_ascii_decimal` which is intended for these cases.
Reviewed By: WillerZ
Differential Revision: D27281875
fbshipit-source-id: 0c98749e4aed9c873853eed2221cf54a89279ff4
Summary:
The hashes that are passed in as parameters to the hash-to-location function
may not be hashes that actually exist. This change updates the code so that
we don't return an error when an unknown hash is passed in. The unknown
hash will be skipped in the list of results.
Reviewed By: quark-zju
Differential Revision: D27526758
fbshipit-source-id: 8bf9b7a134a6a8a4f78fa0df276f847d922472f5
Summary:
We want to handle the case where the client has multiple heads for master. For
example when master is moved backwards (or when it get moved on the client by
force). Updating the client code to thread the list of master commits to the
EdenApi client.
Reviewed By: quark-zju
Differential Revision: D27523868
fbshipit-source-id: db4148a3f1d0e8b0b162e0ecc934e07f041c5511
Summary:
We want to handle the case where the client has multiple heads for master. For
example when master is moved backwards (or when it get moved on the client by
force). Updating the request object for HashToLocation to send over all the
master heads.
When the server builds non-master commits, we will want to send over non-master
heads too. We may consider having one big list of heads but I think that we
would still want to distinguish the non-master commit case in order to optimize
that use-case.
Reviewed By: quark-zju
Differential Revision: D27521778
fbshipit-source-id: cc83119b47ee90f902c186528186ad57bf023804
Summary:
This scenario appears when master moves backwards. Since the master group in
segmented changelog is append-only. Non-fast-forward master move will cause
multiple heads in the master group.
Since Segmented Changelog was updated to handle multiple master heads, we can
propagate the full list that we get from the client.
This diff makes the assumption that Mononoke will know to convert all client "master head"
hashes from HgChangesetId (Sha1) form to ChangesetId (Blake2). If any of the master
heads cannot be converted then it means the server might not be reliably answer the
client's question (in "ancestors(master_heads)", translate "this hash" to a path, or tell me
confidently that the "hash" is outside "ancestors(master_heads)"). That's an error case.
Reviewed By: quark-zju
Differential Revision: D27521779
fbshipit-source-id: 219e08a66aac17ac06d2cf02676a43c7f37e8e26
Summary:
This scenario appears when master moves backwards.
Since the IdDag can handle multiple master heads, the server can piggy-back on that
functionality and support multiple master heads when translating location to hash.
Reviewed By: quark-zju
Differential Revision: D27521780
fbshipit-source-id: c27541890d4fda13648857f010c11a25bf96ef67
Summary:
Fixes needed:
- buck2/linter commented out - unstable API change
- hermetic/reverie - ExitStatusExt sealed (fixed in D27637862)
- Iterator::fold_first stabilized as reduce
- panic!() requires literal format string (fixed in D27672891)
I didn't change anything under scripts/ - a lot of it looked like it was already broken.
Reviewed By: dtolnay
Differential Revision: D27654989
fbshipit-source-id: d1eac6e51686dfb74c87cec8718f075066df4b8f
Summary:
`panic!()`, and things which use `panic!()` like `assert!()`, take a literal format
string, and whatever parameters to format. There's no need to use `format!()`
with it, and it is incorrect to pass a non-literal string.
Mostly it's harmless, but there are some genuinely confusing asserts which
trigger this warning.
Reviewed By: dtolnay
Differential Revision: D27672891
fbshipit-source-id: 73929cc77c4b3d354bd315d8d4b91ed093d2946b
Summary:
Modify the `Debug` implementation for `minibytes::Bytes` to use `std::ascii::escape_default` to debug print a `Bytes` as an ascii-escaped bytestring.
For comparison, the `bytes` crate `Bytes` type provides the same functionality, though it doesn't use the standard library `escape_default` function: https://docs.rs/bytes/1.0.1/src/bytes/fmt/debug.rs.html#39-43
This change greatly improves the output of the `debugscmstore` command. If we don't want to make this the default behavior, we can provide a formatting wrapper type or I can specialize the output in `debugscmstore`, but I can't see any real downsides, especially given the `bytes` crate does the same thing, and we have a similar specialization for `HgId` (hex format in that case).
Reviewed By: quark-zju
Differential Revision: D27642721
fbshipit-source-id: 8faba421fa5082a2098b13ef7d286e05eccb6400
Summary: Add the `with_key` function to `Entry`, which replaces it's key with a provided key. Currently, scmstore returns incorrect results when multiple entries exist with different paths but the same HgId (as scmstore directly returns the path found on disk locally). This isn't a problem in the legacy API, which returns a bare `Vec<u8>` content, which is implicitly associated with the requesting key because it is the result of a single `get` call, or is irrelevant because the `prefetch` method doesn't directly return the results.
Reviewed By: andll
Differential Revision: D27664025
fbshipit-source-id: 014d44ca9a1dc2721685622fd2b077ed3483838f
Summary:
We are moving our skelton for eden top a bit further, by creating the view
structure for eden top. This creates the widget for the banner of eden top,
the help page and the main page as well as a couple sections of the main
page.
The main page is displayed on default and they are toggled with `h` for help
page and `esc` to return to the mage page. Visable and hidden widgets are not
implemented in termwiz yet so we have to do a bit of hacking to hide and
display widgets for our selves.
Each of the sections is stubed with place holder text for testing.
Reviewed By: xavierd
Differential Revision: D26892620
fbshipit-source-id: a7bb4d0e11f3a8968ef071e7f585d07a9c286880
Summary:
D27659634 (8e8aaa61d6) removed these files, so let's drop their exclusions from
test-check-code.t
Reviewed By: sfilipco
Differential Revision: D27682136
fbshipit-source-id: f8e10fac37ea90fb2782b960faf4536f1ff9133b
Summary:
fctx is not guaranteed to have the _path and _filenode attributes. Those are
specific to implementations, e.g. `absentfilectx` does not have them.
`basefilectx` instead defines the `path()` and `filenode()` for general fctx
use.
Reviewed By: quark-zju
Differential Revision: D27667176
fbshipit-source-id: 1d7889d264b597665ef05f84a752323f078cb455
Summary:
Create a fork of the Mercurial code that we can use to build server
rpms. The hg servers will continue to exist for a few more months while we move
the darkstorm and ediscovery use cases off them. In the mean time, we want to
start making breaking changes to the client, so let's create a stable copy of
the hg code to produce rpms for the hg servers.
The fork is based off c7770c78d, the latest hg release.
This copies the files as is, then adds some minor tweaks to get it to build:
- Disables some lint checks that appear to be bypassed by path
- sed replace eden/scm with eden/hg-server
- Removed a dependency on scm/telemetry from the edenfs-client tests since
scm/telemetry pulls in the original eden/scm/lib/configparser which conflicts
with the hg-server conflict parser.
allow-large-files
Reviewed By: quark-zju
Differential Revision: D27632557
fbshipit-source-id: b2f442f4ec000ea08e4d62de068750832198e1f4
Summary:
as titled, an ugly hack to support LABEL attribute for 1.7.0 which didn't exist.
getting rid of it now since oss support upgraded to 1.7.1
Differential Revision: D27536730
fbshipit-source-id: 58653e71cbf20a1cda0b7414f66b5f0014d89b84
Summary:
Sometimes we can hit an Idle timeout error while talking to MySQL, because we open a connection and go idle for a long time. Then when we finally send a query, server returns an error: the connection is expired. This is the issue we found and fixed in D27503062 (a856799489) that blocked MySQL client release.
## Future starvation
Imagine you have a stream in which you're connecting to a server, fetching and preparing some values:
```
let v = vec![1u32, 2, 3, 4, 5];
let mut s = stream::iter(v)
.map(|key| async move {
let conn = connect(..).await?;
conn.fetch(..).await
})
.buffered(2)
.map(|item| async move { prepare(..) })
.buffered(2);
```
Now you want to asynchronously process those prepared values one by one:
```
while let Some(result) = s.next().await {
let value = result?;
process(value).await?;
}
```
This async `process(..)` call can be talking to some service to take these values or something else that doesn't require much of a CPU time. Although the operation can be long.
**Now what happens when we do s.next().await?**
Because the stream is `buffered(2)` we wait for the first 2 futures. When the first item is ready, it returns the result and polls next stream item with a key - 3. The third future only makes a `connect(..)` call and gets switched.
When we've got a next value from the stream, we're waiting on the `process(value)` call and not polling the underlying stream till the processing is done.
**As I mentioned earlier, it is not expensive...**
But what if it takes > 10s to complete anyway?
The third future from the stream, that was polled earlier, **will wait for all these > 10s till it is polled again**.
More details [in this post](https://fb.workplace.com/groups/learningrust/permalink/2890621307875402/).
## Solution
In this case spawning a future with connection and query steps is a way to fix the issue.
This diff spawns queries in `shed::sql::sql_common::mysql_wrapper` - this covers all the places in Mononoke where we talk to MySQL. Also I removed the spawn from hg sync code, because it is not needed anymore and to illustrate that this approach works.
Reviewed By: StanislavGlebik
Differential Revision: D27639629
fbshipit-source-id: edaa2ce8f5948bf44e1899a19b443935920e33ef
Summary:
`getdeps` builds sometimes fail - try a higher compression level to
see if it's a different default internally and in open source
Reviewed By: ahornby
Differential Revision: D27659420
fbshipit-source-id: 702341e6b288ab79584bfa8de5b1ccd5ed6bc57a
Summary:
Somehow, rmtree can fail to open the file being removed, in which case the
function is `os.open` that requires a flag on top of the path. When that
happens a `TypeError` will be raised:
```
✗ Fail: eden/integration:integration - test_eden_start_fails_if_edenfs_fails_during_startup (eden.integration.start_test.StartFakeEdenFSTestSystemdEdenCLI) (4.363)
Test output:
> TypeError: open() missing required argument 'flags' (pos 2)
File "/usr/local/fbcode/platform009/lib/python3.8/unittest/case.py", line 60, in testPartExecutor
yield
File "/usr/local/fbcode/platform009/lib/python3.8/unittest/case.py", line 717, in doCleanups
self._callCleanup(function, *args, **kwargs)
File "/usr/local/fbcode/platform009/lib/python3.8/unittest/async_case.py", line 72, in _callCleanup
self._callMaybeAsync(function, *args, **kwargs)
File "/usr/local/fbcode/platform009/lib/python3.8/unittest/async_case.py", line 84, in _callMaybeAsync
ret = func(*args, **kwargs)
File "/usr/local/fbcode/platform009/lib/python3.8/contextlib.py", line 533, in close
self.__exit__(None, None, None)
File "/usr/local/fbcode/platform009/lib/python3.8/contextlib.py", line 525, in __exit__
raise exc_details[1]
File "/usr/local/fbcode/platform009/lib/python3.8/contextlib.py", line 510, in __exit__
if cb(*exc_details):
File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/aab7ed39/eden/integration/integration#binary,link-tree/eden/test_support/temporary_directory.py", line 104, in __exit__
self.cleanup(exc_type is not None)
File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/aab7ed39/eden/integration/integration#binary,link-tree/eden/test_support/temporary_directory.py", line 117, in cleanup
cleanup_tmp_dir(temp_dir)
File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/aab7ed39/eden/integration/integration#binary,link-tree/eden/test_support/temporary_directory.py", line 71, in cleanup_tmp_dir
shutil.rmtree(tmp_dir, onerror=_remove_readonly)
File "/usr/local/fbcode/platform009/lib/python3.8/shutil.py", line 715, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/usr/local/fbcode/platform009/lib/python3.8/shutil.py", line 652, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File "/usr/local/fbcode/platform009/lib/python3.8/shutil.py", line 652, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File "/usr/local/fbcode/platform009/lib/python3.8/shutil.py", line 652, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File "/usr/local/fbcode/platform009/lib/python3.8/shutil.py", line 648, in _rmtree_safe_fd
onerror(os.open, fullname, sys.exc_info())
File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/aab7ed39/eden/integration/integration#binary,link-tree/eden/test_support/temporary_directory.py", line 66, in _remove_readonly
func(path)
```
To bypass this, let's simply test if the function is os.unlink or os.rmdir
first.
Reviewed By: chadaustin
Differential Revision: D27654126
fbshipit-source-id: 318c547f363c020e98e21d5f07a9c7a9b2e5102d
Summary:
Our logs are filled with this message, making it almost impossible to read,
this doesn't need to be DBG1, let's reduce it to DBG5 similarly to what we do
for blobs.
Reviewed By: kmancini
Differential Revision: D27652526
fbshipit-source-id: c61d502a9f17d8180bad857ff6c1a9033763947d
Summary:
Those are not used. Recently we saw build issues like:
lib/third-party/sha1dc/sha1.c:8:10: fatal error: string.h: No such file or directory
#include <string.h>
^~~~~~~~~~
Possibly by some compiler flags disabling stdlib. Since we don't need
the C code let's just remove them.
Reviewed By: singhsrb
Differential Revision: D27659634
fbshipit-source-id: b8fcac4f05be95cac470f9b4830baf76e06f98ad
Summary:
Defend against the possibility of `nodeinfo["remote_bookmarks"]` being `None`
by treating it the same as an empty list.
Reviewed By: quark-zju
Differential Revision: D27658994
fbshipit-source-id: 06c31bdba2cc6cbd82f5bd718befd6be9bb233f2
Summary:
The `python` in `PATH` might be the fbcode Python that has `-nostdinc` in its
`CFLAGS`. That Python is problematic because it cannot compile C extensions.
Reviewed By: DurhamG, markbt
Differential Revision: D27661488
fbshipit-source-id: 243205522fcaf53d5af6a3c9afc4d28160072de5
Summary:
a [git worktree](https://git-scm.com/docs/git-worktree) allows for a single repository to have multiple local checkouts in different directories (allowing parallel work on multiple branches of the same repo).
in such a worktree, `.git` is a file and not a directory:
```
$ cat .git
gitdir: /path/to/actual/repo/.git/worktrees/name
```
the `_git_prompt` helper in `scm-prompt.sh` expects only directories:
```
$ echo "$(_git_prompt /path/to/worktree/.git)"
# (output is empty)
```
this amends the helper to support them:
```
$ echo "$(_git_prompt /path/to/worktree/.git)"
f140443f|REBASE-i|T123456
$ echo "$(_git_prompt /path/to/parent/.git)"
T54321
```
Reviewed By: quark-zju
Differential Revision: D27509147
fbshipit-source-id: 9a4ab55af99538c6425ad5721c3266386dbda453
Summary:
The "filter" set's filter function might not be prepared for inputs outside
the parent set. So let's the "contains" function to test against the parent
set first, then test the filter function.
This fixes the "merge()" set's "contains" check using the revlog backend:
In [1]: v=repo.revs('draft() & merge()')
In [2]: v._set
Out[2]: <meta ?>
In [3]: m.node.nullid in v._set
Out[3]: False
Before this diff it would be:
In [3]: m.node.nullid in v._set
CommitLookupError: '0000000000000000000000000000000000000000 cannot be found'
Note: Segmented changelog backend is not affected because it does not use
filter sets.
Reviewed By: xavierd
Differential Revision: D27657502
fbshipit-source-id: 30bb261fea59bdc5644580e98796f52fa93c2705
Summary: Update the `curl` and `curl-sys` crates to use a patched version that supports `CURLOPT_SSLCERT_BLOB` and similar config options that allow the use of in-memory TLS credentials. These options were added last year in libcurl version `7.71.0`, but the Rust bindings have not yet been updated to support them. I intend to upstream this patch, but until then, this will allow us to use these options in fbcode.
Reviewed By: quark-zju
Differential Revision: D27633208
fbshipit-source-id: 911e0b8809bc0144ad8b32749e71208bd08458fd