Commit Graph

12499 Commits

Author SHA1 Message Date
Jun Wu
a50aab4407 dag: remove tracing-futures dep
Summary: It is no longer necessary.

Reviewed By: andll

Differential Revision: D31358737

fbshipit-source-id: 02d54ab119599c1b4f9bf887840d282db6a08b99
2021-10-12 14:17:39 -07:00
Pyre Bot Jr
dacf399ba2 suppress errors in fbcode/eden - batch 1
Differential Revision: D31579806

fbshipit-source-id: a5992b27796c6785c4ff6e59b5c8c97ecec7aef2
2021-10-12 13:08:00 -07:00
Katie Mancini
864a6c1c1a make eden removal on windows a bit more graceful
Summary:
When eden is not running on windows `eden rm` will spit out some error
messages and exits with error code 1. But it does actually successfully
remove the repo.

On linux removing a repo while eden is not running behaves just like if
eden were running.

Let's make the removal more graceful on windows.

Reviewed By: xavierd

Differential Revision: D31519805

fbshipit-source-id: d393922a9474e64251142207ae38a602766f17bf
2021-10-12 10:22:05 -07:00
Alex Hornby
b7c6b1d17d mononoke: use auto_impl Arc for synced_commit_mapping
Summary: Removes some boiler plate and implements the trait more consistently for Arc<X> as well as Arc<dyn X>

Reviewed By: Croohand

Differential Revision: D31558567

fbshipit-source-id: 503e745ccf4eba6dcf03b8a3f4ace49310c1c319
2021-10-12 10:14:46 -07:00
Stanislau Hlebik
78b7074edd mononoke: remove unused pushrebase_changegroup_part
Summary: It's unused.

Reviewed By: yancouto

Differential Revision: D31573897

fbshipit-source-id: 3fdb1a2bf7bcc13fda024a7be42a3708ae26825e
2021-10-12 09:01:49 -07:00
Alex Hornby
d2dcc7174b mononoke: use create if not exists for sqlite
Summary: This simplifies test setup and means can stop throwing away creation SQL errors

Reviewed By: StanislavGlebik

Differential Revision: D31536408

fbshipit-source-id: 4d3df0f69c5b49719196c8a1b20d65c965d88869
2021-10-12 06:46:58 -07:00
Thomas Orozco
977755e59e Update autocargo component on FBS:master
Summary:
Automated component version update
Bump Schedule: https://www.internalfb.com/intern/msdk/bump/?schedule_fbid=342556550408072
Package: https://www.internalfb.com/intern/msdk/package/125803836415945/
Oncall Team: rust_foundation
NOTE: This build is expected to expire at 2022/10/11 06:06AM PDT
---------
New project source changes since last bump based on 9cc37ee9d47923093119b37d4f2d60de5a5e490f at 2021/09/21 01:03PM UTC:
| 2021/09/21 01:05PM -05 | generatedunixname89002005294178 | D31082067 | [MSDK] Update autocargo component on FBS:master |
| 2021/09/29 01:58PM PDT | vgao1996 | D31115820 (ae87b82eaf) | [Rust] update rand and quickcheck |
| 2021/09/30 02:55PM PDT | jkeljo | D31284743 (542e84d8fc) | [rust][third-party] Enable `unbounded_depth` feature for `serde_json` |
| 2021/10/08 11:43AM BST | krallin | D31471849 | autocargo: emit rerun-if-changed on the buildscript itself |
---------

build-break (bot commits are not reviewed by a human)

Reviewed By: farnz

Differential Revision: D31541798

fbshipit-source-id: 8e41a39d552c31a879c41b7d244e2caca047b7da
2021-10-12 04:20:53 -07:00
Xavier Deguillard
0357bb24c1 integration: use sys.executable to run edenfsctl, not sys.platform
Summary: sys.platform is "win32", not the path to python.

Reviewed By: fanzeyi

Differential Revision: D31554106

fbshipit-source-id: 64b388d2fb8a493f811a0cf22fe2471a25bfbf7e
2021-10-11 17:28:34 -07:00
Muir Manders
88392b71a3 config: close race condition writing hgrc.dynamic
Summary: Use atomic rename instead of fs::write when writing out the hgrc.dynamic config file. fs::write re-writes the file if it already exists which opens a race condition if two hg processes try to write the config at the same time.

Reviewed By: DurhamG

Differential Revision: D31548054

fbshipit-source-id: c5216bf6f9f2ff78d37a2c0e23c8c22fdf1d0897
2021-10-11 17:15:33 -07:00
Muir Manders
a7c5f453c5 rebase: maintain order when printing commit mappings
Summary: Rebase (and other commands) print the mappings between old and new commits via the tweakdefaults extension. Previously we would sort the node mapping dictionary which yields apparent random ordering. Now instead we maintain dictionary order which by default gives us the dictionary insert order (which in rebase's case, at least, is revision order).

Reviewed By: DurhamG

Differential Revision: D31551290

fbshipit-source-id: 0b5d9c4846ba42154ae387bdf4777a61e5e1e605
2021-10-11 17:10:29 -07:00
Stanislau Hlebik
93e2be0504 mononoke: second attempt to fix deadlock
Summary:
I've investigated the deadlock we hit recently and attempted to fix it with D31310626 (8ca8816fdd).
Now there's a bunch of facts, an unproven theory and a potential fix. Even though the theory is not proven and hence the fix might be wrong, the theory sounds plausible and the fix should be safe to land even if the theory is wrong.

But first, the facts,

### Facts

a)  There's a repro for a deadlock - running `buck-out/gen/eden/mononoke/git/gitimport/gitimport#binary/gitimport --use-mysql-client --repo-name aosp --mononoke-config-path configerator://scm/mononoke/repos/tiers/prod --panic-fate=exit --cachelib-only-blobstore /data/users/stash/gitrepos/boot_images git-range 46cdf6335e1c737f6cf22b89f3438ffabe13b8ae b4106a25a4d8a1168ebe9b7f074740237932b82e` very often deadlocks (maybe not every time, but at least every 3rd run). So even though D31310626 (8ca8816fdd) unblocked the mega_grepo_sync, it doesn't seem like it fixed or even mitigated the issue consistently. Note that I'm running it on devbig which has quite a lot of cpus - this may or may not make deadlock more likely.

b) The code deadlocks on [`find_file_changes()`](https://fburl.com/code/7i6tt7om) function, and inside this function we do two things - first we find what needs to be uploaded using [`bonsai_diff()`](https://fburl.com/code/az3v3sbk) function, and then we use [`do_upload()`](https://fburl.com/code/kgb98kg9) function to upload contents to the mononoke  repo. Note that both `bonsai_diff()` and `do_upload()` use git_pool. Bonsai_diff produces a stream, and then we apply do_upload to each element of the stream and finally we apply [`buffer_unordered(100)`](https://fburl.com/code/6tuqp3jd) to upload them all in parallel.

In simplified pseudo-code it looks like
```
bonsai_diff(..., git_pool, ...)
  .map(|entry| do_upload(..., git_pool, entry, ....)
  .try_buffer_unordered(100)
```

c) I've added a few printlns in P462159232, and run the repro command until it, well, repros. These printlns just shows when there was an attempt to grab a semaphore and when it successfully grabbed, and also who was the caller - `bonsai_diff()` or `do_upload()` function. This is the example output I'm getting P462159671. The output is a mess, but what's interesting here is that there exactly 100 more entries of "grabbing semaphore uploading" than "grabbed semaphore uploading". And 100 is exactly the size of the buffer in buffer_unordered.

### Theory
So above are the facts, and below is the theory that fits these facts (but as I said above, the theory is unproven yet). Let me start with a few assumption

1) Fact `c)` seem to suggest that when we run into a deadlock there was a full buffer of `do_upload()` futures that were all waiting on a semaphore.
1) If we look at [`buffer_unordered` code](https://docs.rs/futures-util/0.3.17/src/futures_util/stream/stream/buffer_unordered.rs.html#71) we can see that if buffer is full then it stops polling underlying stream until any of the buffered futures is done.
1) Once semaphore is released it [wakes up just a handful of futures](https://docs.rs/tokio/1.12.0/src/tokio/sync/batch_semaphore.rs.html#242) that wait for this semaphore.

Given this assumptions I believe the following situation is happening:
1) We get into a state where we have 100 `do_upload()` futures in buffer_unordered stream all waiting for a semaphore. At the same time all the semaphore permits are grabbed by `bonsai_diff()` stream (because of assumption #1)
2) Because of assumption #2 we don't poll underlying `bonsai_diff()` stream. However this stream has already [spawned a few tasks](https://fburl.com/code/9iq3tfad) which successfully grabbed the semaphore permit and are executing
3) Once these `bonsai_diff()` tasks are finished they [drop the semaphore permit](https://fburl.com/code/sw5fwccw), which in turn causes semaphore to be released and one of the tasks that are waiting to be woken up (see assumption #3). But now two things can happen - either a `do_upload()` future will be woken up or `bonsai_diff()` future will be woken up. If the former happens then `buffer_unordered` would poll the `do_upload()` future and continue executing. However if the latter happens (i.e. `bonsai_diff()` future was woken up) then it causes a deadlock - buffer_unordered() aren't going to poll `bonsai_diff()` stream, and so bonsai_diff() stream won't be able to make any progress. At the same time `do_upload()` futures aren't going to be polled at all because they weren't woken up in the first place, and so we deadlock.

There are a few things that seem unlikely in this theory (e.g. why we never wake up any of the `do_upload()` futures), so I do think it's worth validating. We could either add a bunch of println! in buffer_unordered and semaphore code, or try to setup [tokio-console](https://github.com/tokio-rs/console).

### Potential fix

Potential fix is simple - let's just add another spawn around `GitPool::with()` function. If the theory is right, then this fix should work. Once a semaphore is released and a `bonsai_diff()` future is woken up then this future will be spawned and hence will be able to complete successfully regardless of whether `buffer_unordered()` has its buffer full or not. And once `bonsai_diff()` fully completes then finally `do_upload()` futures will be woken up and `buffer_unordered()` will progress until completion.

If the theory is wrong then we just added one useless `tokio::spawn()`. We have a lot of spawns all over the place and `tokio::spawn` are cheap. So landing this fix shouldn't make things worse, but it might actually improve them (and the testing I did suggests that it indeed should improve them - see `Test plan` section below). I don't think it should be seen as a reason to not find the root cause

Reviewed By: mojsarn

Differential Revision: D31541432

fbshipit-source-id: 0260226a21e6e5e0b41736f86fb54a3abccd0a0b
2021-10-11 14:57:28 -07:00
Zhaolong Zhu
cebe35eb7b Add eden prefetch-profile list output to eden rage
Summary: Add `eden prefetch-profile list` output to `eden rage`

Reviewed By: genevievehelsel

Differential Revision: D31488296

fbshipit-source-id: 216686dba842b8c5cec283627178d3e8c8b2a7e8
2021-10-11 11:21:51 -07:00
Dave Kwon
c344a5629e Update thrift_format component on FBS:master
Summary:
Incorporated Diffs:
* Fix blank lines between structured annotated fields D31372368
* Fix format for empty struct D31489644

Manual component version update
Bump Schedule: https://www.internalfb.com/intern/msdk/bump/?schedule_fbid=326075138621961
Package: https://www.internalfb.com/intern/msdk/package/608249566967196/
Oncall Team: nuclide
NOTE: This build is expected to expire at 2022/10/08 03:04PM PDT
---------
New project source changes since last bump based on D30698375 (e5f5b589b458e473ab300550737172d43167f2bf at 2021/09/01 04:03PM IST):
no related diffs found between since last bump
---------

build-break (bot commits are not reviewed by a human)

Reviewed By: zertosh

Differential Revision: D31517842

fbshipit-source-id: 9f04c0ad1f418ba8ec3c96b05366683413d93f97
2021-10-11 00:19:37 -07:00
Andrei Ilyashenko
639d7b35f0 Bump 'structopt' to from 0.3.21 to 0.3.23
Summary: Bumping structopt version

Reviewed By: MarcoMontaltoMonella

Differential Revision: D31432560

fbshipit-source-id: cd58f374e3039e21ee9b36825b6f4a4915cb94b5
2021-10-09 09:34:57 -07:00
Xavier Deguillard
0fdf2cceec integration: only run edenfsctl.real with python on Windows
Summary: This appears to have broken all the tests on Linux.

Reviewed By: zhengchaol

Differential Revision: D31505082

fbshipit-source-id: 610eb941d0f0eb536a4688ac2637a8466be0b05c
2021-10-08 10:52:51 -07:00
Katie Mancini
05806120f5 make eden top xterm-color friendly
Summary:
xterm-color terminals don't support hiding the cursor the way curses likes to.
this crashes eden top for users.

eden top mostly functions fine otherwise in xterm-color, so let's make
this a non fatal error.

Note: the highlighting behavior is a little wierd (the styling after the
highlighted column does not work, no bolding etc). This looks like a wierd
curses bug on these terminals, so not sure if we can really fix this. I figure
a little wierd styling is still better than a crash.

Reviewed By: fanzeyi

Differential Revision: D31480121

fbshipit-source-id: 581ef7c548fd1f7986c4f93d8b797d7f7588c351
2021-10-08 09:43:02 -07:00
Jan Mazur
19b6586010 add clientinfo.u64 tag to identify clients in logs
Summary: This will let us identify clients in scuba logs. This will be useful to identify whether a client has some specific feature enabled. We can use this instead of dedicated boolean columns like is_eden, or is_using_feature_X.

Reviewed By: krallin

Differential Revision: D31501373

fbshipit-source-id: 0e63b73659fd145f01098d60ced510e464730982
2021-10-08 08:37:30 -07:00
Harvey Hunt
636d71b38a mononoke: Parse clientinfo and log sandcastle data to scuba
Summary:
The hg client provides a header which contains `ClientInfo` data. This
includes the sandcastle alias and nonce.

Update Mononoke to parse this header and then log the sandcastle alias and
nonce to scuba.

Reviewed By: krallin

Differential Revision: D31208450

fbshipit-source-id: f0971b668887de47fbab29b7ce9b0173cbdeafe4
2021-10-08 08:00:57 -07:00
Harvey Hunt
6e2ec377d9 mononoke: Refactor metadata creation
Summary:
The `Metadata::new` function has grown quite a few optional arguments
which can make it difficult to read.

Simplify the `new` method and add new methods to record optional information
(such as encoded CATs).

Reviewed By: krallin

Differential Revision: D31500788

fbshipit-source-id: 9675c39f3061fef614e792e6ab6e365e9b423b2a
2021-10-08 08:00:57 -07:00
Harvey Hunt
83e9a77d06 mononoke: lfs: Log clientinfo to scuba
Summary:
The hg client provides extra info in a header that now includes the
sandcastle alias and nonce.

Update the LFS server to parse the header and log the sandcastle alias and
nonce to scuba.

Reviewed By: krallin

Differential Revision: D31208449

fbshipit-source-id: 773f0ec22060203c2c74a20090bd4893885506f8
2021-10-08 08:00:57 -07:00
Harvey Hunt
0173e9176d hg: Use latest version of reqwest
Summary:
The hg client uses a version of reqwest that relies on tokio-0.2. In
the next diff I will add `clientinfo` as a dependency of Mononoke LFS server,
which also pulls in `configparser`. However, Mononoke LFS server uses
tokio-1.0.

Update the reqwest dependency to be the latest version.

Reviewed By: quark-zju

Differential Revision: D31389559

fbshipit-source-id: acf4c3b5c9df2a8bc8cfa134a2d314fbb96c3354
2021-10-08 08:00:57 -07:00
Jan Mazur
a47683ab56 debugnetwork will use x2pagentd proxy when available (but not on Windows)
Summary: If regular traffic goes through the agent, debug traffic should as well.

Differential Revision: D31500581

fbshipit-source-id: 32abef4e082dbf120c9aa104206b460e12ed506f
2021-10-08 07:33:23 -07:00
Jan Mazur
42232c29e4 edneapi on x2pagentd (and everything using hg-http)
Summary: this should enable everything using rust's http client to go over x2pagentd. The biggest user is edenapi. Should work on all platforms (including Windows!).

Reviewed By: ahornby

Differential Revision: D31430275

fbshipit-source-id: f90a633eb3cc4e82447b1b76200499016dcb6b8e
2021-10-08 06:45:39 -07:00
Jan Mazur
361b61676a sync RepoConfigs configerator thrift structure to fbcode
Summary: I just landed D30899990. I'm landing this to sync the config, still without using it.

Differential Revision: D31472892

fbshipit-source-id: 5e18c4c3529118ef81880c886f0d8b9428efcbf4
2021-10-08 05:33:38 -07:00
Zeyi (Rice) Fan
a56385f570 fix facebook:enable-service-router default value
Summary: `eden-config.h` was not included previously and caused the preprocessor macro to always use the default value false path.

Reviewed By: chadaustin

Differential Revision: D31462811

fbshipit-source-id: ade236ce1f5b2b6511163515ced79890855190f2
2021-10-07 12:21:06 -07:00
Xavier Deguillard
f7a3db7aaf integration: run edenfsctl.real with python
Summary:
On Windows, edenfsctl.real is a par file that Windows can't execute directly.
Thus let's have Python run it.

Reviewed By: fanzeyi

Differential Revision: D31477721

fbshipit-source-id: d5a699ceb3d3b1b3d5778ef5720bca7c299bed80
2021-10-07 12:06:18 -07:00
Xavier Deguillard
9f59be4b6f integration: fix redirect tests on Windows
Summary:
These tests have been failing for a long time due to the expected output on
Windows containing \ but the test using the / separator.

Let's simply use os.path.join to build a path with the right separator in these
tests.

Reviewed By: fanzeyi

Differential Revision: D31477722

fbshipit-source-id: a4ac25750647229974e23e305508e83917011fef
2021-10-07 12:06:18 -07:00
Mateusz Kwapich
666edf39d8 increase the number of attemps on import
Reviewed By: krallin

Differential Revision: D31441096

fbshipit-source-id: 852500de99dd2c7e2ccc2f3ba5002795a7226996
2021-10-07 04:51:39 -07:00
Egor Tkachenko
0656f26058 Make CoreContext::test_mock log in unit-tests
Summary:
Current `test_mock` creates logger with default_drain and output isn't printed in unit-tests. There is a method `logger_that_can_work_in_tests()` which overcome it, let's use it for constructing test CoreContext.

While testing I've also spotted some leftover from migration to new BonsaiDerivable. Lets fix it too

Reviewed By: krallin

Differential Revision: D31430162

fbshipit-source-id: a086be521f0ceaeb3267e87f24980fb11587a6e7
2021-10-07 04:08:22 -07:00
Xavier Deguillard
81da058470 inodes: make TreeInode::removeImpl use ImmediateFuture
Summary:
The callers are still converting from an ImmediateFuture to a folly::Future,
this will be tackled in a subsequent diff.

Reviewed By: fanzeyi

Differential Revision: D31349658

fbshipit-source-id: f3cef52d3fb4b6c1fb0af73399a57e8d163216b8
2021-10-06 17:18:25 -07:00
Xavier Deguillard
21cfa267a2 inodes: constify GlobNode::evaluate
Summary: Thes methods don't need to be non-const, thus let's constify them.

Reviewed By: chadaustin

Differential Revision: D31344890

fbshipit-source-id: 3f0ac7615993530626badced45debfd2ac91d8a7
2021-10-06 17:18:25 -07:00
Xavier Deguillard
4f9bfb66fc inodes: avoid unecessary shared_ptr copies
Summary:
Passing shared_ptr by copy everywhere can be expensive as it forces an atomic
operation to be performed. Since the caller of the glob code can easily
guarantee that the data will outlive the globbing code, let's just pass
references/pointers to it so that only references and pointers are copied.

Reviewed By: genevievehelsel

Differential Revision: D31344889

fbshipit-source-id: cee797202470aa123381d9ee22e11780722f5b33
2021-10-06 17:18:25 -07:00
Jun Wu
9247ce318a edenapi: add retries for hash<->location endpoints
Summary: The endpoint can fail sometimes. Add retries to make it more reliable.

Reviewed By: yancouto

Differential Revision: D31407279

fbshipit-source-id: 1b05feedd65477aa0b9cd07be30217b4f2e6d1b2
2021-10-06 17:14:04 -07:00
Jun Wu
a0a58d77d4 http-client: add Clone for some structs
Summary: This will be used by the next change.

Reviewed By: yancouto

Differential Revision: D31407281

fbshipit-source-id: 6e3e26a0fb4864d2076d2aafcb0f0ba919bfe2cc
2021-10-06 17:14:04 -07:00
Jun Wu
b68ef5e195 edenapi: change hash<->location from Response<T> to Vec<T>
Summary:
The `Response` type in edenapi has extra complexity about async streaming.
It is more useful for large data like file contents. For metadata like the
hash<->location translation, using a plain `Vec` is much simpler. Using
`Vec` also makes it easier to implement retry logic.

Reviewed By: yancouto

Differential Revision: D31407278

fbshipit-source-id: 9d6df225d183eaffe72f99cfd53ae3cd2987a518
2021-10-06 17:14:04 -07:00
Jun Wu
b16b09239e edenapi: remove EdenApiBlocking
Summary:
We concluded that ideally `sync -> block_on(async)` only happens in Python
wrappers, not pure Rust code. With that, `EdenApiBlocking` makes it easier
to write anti-pattern pure Rust code, and itself looks a bit repetitive.
All users of `EdenApiBlocking` have been migrated. So let's drop
`EdenApiBlocking` now.

Reviewed By: yancouto

Differential Revision: D31407284

fbshipit-source-id: 797506ccd07f413ac041dcd2ea07b3a3519d912f
2021-10-06 17:14:04 -07:00
Jun Wu
cbb7df0fc1 edenapi: avoid EdenApiBlocking in C bindings
Summary: Unblocks removing `EdenApiBlocking`.

Reviewed By: yancouto

Differential Revision: D31416612

fbshipit-source-id: c2aa1add69c24ba247c4bdcfcacaeaceb15ac0d6
2021-10-06 17:14:04 -07:00
Jun Wu
c22c44397e pyedenapi: avoid EdenApiBlocking
Summary:
There are only a few places using EdenApiBlocking. The other endpoints are just
using explicit `block_on` constructs.

Let's migrate the endpoints off the EdenApiBlocking trait.

Reviewed By: yancouto

Differential Revision: D31407287

fbshipit-source-id: edeb16a1ed4f50cc01c75546df7362673f941d01
2021-10-06 17:14:04 -07:00
Jun Wu
1b487cf434 hgcommands: drop EdenApiBlocking dep for debughttp
Summary:
EdenApiBlocking is going away. Drop another dependency on it.
This also adds proper Ctrl+C support for the command.

Reviewed By: yancouto

Differential Revision: D31407285

fbshipit-source-id: 6c5f40f2933fdd54db197ae6b93c5f240eda1c25
2021-10-06 17:14:04 -07:00
Jun Wu
05139b453b commands: avoid EdenApiBlocking in debugsegmentclone
Summary:
This makes it possible to drop EdenApiBlocking. Since the actual lazy clone
logic is in the Python code, we might want to only keep one of the Rust or
Python lazy clone logic eventually.

Reviewed By: yancouto

Differential Revision: D31407286

fbshipit-source-id: 3685edbf6e4709aaf8190ba65035d6627ef83fe9
2021-10-06 17:14:04 -07:00
Jun Wu
7b4881a7a4 revisionstore: drop dep on EdenApiBlocking
Summary:
`EdenApiBlocking` was there to provide some convenience for non-async code to
use edenapi. Now a lot of places actually use async Rust properly, and calling
async Rust from non-async Rust is not a great practice. So let's plan for
`EdenApiBlocking` removal. To unblock it, drop `EdenApiBlocking` usage in
revisionstore.

Reviewed By: yancouto

Differential Revision: D31407282

fbshipit-source-id: 70a6e9468606176cf5cf119418547e694e229ec4
2021-10-06 17:14:04 -07:00
Jun Wu
761083b02a edenapi: make retry count configurable
Summary:
Add a config option `edenapi.max-retry-per-request` to control the maximum
retry count. Replace the hard-coded 10 with it.

Reviewed By: yancouto

Differential Revision: D31407280

fbshipit-source-id: 654c79601fe12007e6cbf7ac6a4110da420801dd
2021-10-06 17:14:04 -07:00
Jun Wu
7fec2084fe edenapi: add a general-purposed retry method
Summary: Add a general retry method for futures. Move clone and pull paths to use it.

Reviewed By: yancouto

Differential Revision: D31407283

fbshipit-source-id: 46ec2bd5bacdcd10ae5078ced8d12f2c8b9ed1ec
2021-10-06 17:14:03 -07:00
Jun Wu
c14f3bd128 debugresetheads: make "no head left" error more clear
Summary:
The error is most likely a misconfiguration. Prompt the user to check the
message.

Differential Revision: D31383947

fbshipit-source-id: c58c2b026048266fc0a8c90f31928c97a2381258
2021-10-06 15:29:59 -07:00
Yan Soares Couto
30d4c8fc2d Make RNG in 'arbitrary' calls more correct
Summary:
Using modulo on arbitrary integers to get random numbers [isn't correct](https://www.internalfb.com/diff/D31305392 (da13975a4f)?dst_version_fbid=311037904117090&transaction_fbid=550270779610744), as the distribution between numbers isn't fair (unless the size is a power of two).

This was raised on D31305392 (da13975a4f), but we decided to land that quickly to unblock builds before doing these changes.

I'm applying the changes suggested on D31305392 (da13975a4f). This is what this diff does:
- For all cases where we generate small numbers (up to 5), replace with call to `Gen::choose`, so `u32::arbitrary(g) % 3` becomes `g.choose(&[0, 1, 2]).unwrap()`.
- For generating numbers in range 0..=1, I instead replaced with generating a boolean, which gets rid of the `unreachable!` calls.
- I removed the code to generate numbers in range 0..=0.
- For generating larger numbers, I used `u64::arbitrary` instead, which should make things "less wrong".

Some things I assumed, but am happy to change before landing, just let me know:
- Theoretically we don't *need* to change the code for `% 2` and `% 4`, as the math checks out there. I changed it for consistency there, but am happy to change it back.
- Using boolean also wasn't suggested initially, I'm happy to change back.

Reviewed By: krallin

Differential Revision: D31379381

fbshipit-source-id: a0bac26ebabd32a6c65f717512de998ef5dc37c8
2021-10-06 09:39:23 -07:00
Ilia Medianikov
acca78ca5f mononoke/scs: make commit_lookup_pushrebase_history more precise by using sync direction
Summary:
`commit_lookup_pushrebase_history` is an endpoint that tries to traverse Pushrebase and Commit Sync mappings to find the commit's pushed version for a landed commit. But currently it traverses Commit Sync mapping blindly because it doesn't know the sync direction. This can (very rare) lead to an inaccurate result.

Let's use the `source_repo` column I've introduced in this diff stack and don't traverse in the wrong direction if we know it is wrong.

Reviewed By: StanislavGlebik

Differential Revision: D30975759

fbshipit-source-id: 9c5ecf059dcdebf0c91f0c5545f0c6e95610c2ec
2021-10-06 09:22:33 -07:00
Liubov Dmitrieva
055afa0ca2 modernise tests by using moder newserver function
Summary:
modernise tests by using moder newserver function

the function is shorter and it will also allow us to run the tests with Mononoke in the future

Reviewed By: yancouto

Differential Revision: D30773102

fbshipit-source-id: 994cbcfb2688aef3e96446e1cb021db72bc70c67
2021-10-06 08:14:05 -07:00
Yan Soares Couto
07d0da77d8 Fix quickcheck mincode test
Summary:
Another test that was broken by quickcheck update. I notice it breaking a run of `hg_windows` [here](https://www.internalfb.com/intern/sandcastle/log/?instance_id=4503600123928895&step_id=4503604846860319&step_index=9&name=Run%20cargo%20test#479) (though it's not windows specific).

The problem is quickcheck started using NaN on floats, which surfaced some invalid code. In roundtrip tests, we need our struct to implement `Eq`, which f32 does not (since NaN != NaN). Workaround was implementing something that also considers NaN == NaN.

Question: Do I also fix the hg-server version of the test?

Reviewed By: krallin

Differential Revision: D31401277

fbshipit-source-id: b3eef1a3aef395a1194308788ec74f1bb5a33a42
2021-10-06 07:33:34 -07:00
Mark Juggurnauth-Thomas
8f83d39d17 backfill_derived_data: use background session class
Summary:
The backfiller should use the background session class so that we ensure data
is written to all blobstores in a multiplex.

Reviewed By: yancouto

Differential Revision: D31429168

fbshipit-source-id: 32c767dbef291771565f73cedf3cd01c1a3cce40
2021-10-06 06:31:21 -07:00
Mark Juggurnauth-Thomas
2b6c4ae767 derived_data_manager: mark batch-backfilled commits as derived
Summary:
When rederiving batches of commits, the batch derivation process must mark
these commits as derived so that rederivation will continue with the next
batch.

Reviewed By: yancouto

Differential Revision: D31429169

fbshipit-source-id: e9f6a84a0391ee8d72a0007f39e755410bfac724
2021-10-06 06:31:20 -07:00
Yan Soares Couto
b3b2e16f97 Cleanup old derived data
Summary: Since we migrated all derived data types to use manager, we can now delete a bunch of code.

Reviewed By: krallin

Differential Revision: D31344999

fbshipit-source-id: db864bdc3ba0f95cb34be6e554d629d254f09608
2021-10-06 06:01:42 -07:00
Liubov Dmitrieva
c14d68f17b fix test
Summary: The test output was previously incorrect due to wrong rounding from jq tool

Reviewed By: ahornby

Differential Revision: D31429221

fbshipit-source-id: 2979e393c6f6c1b52e41d732f155275166062bff
2021-10-06 04:40:18 -07:00
Liubov Dmitrieva
d541aea36b fix the values in the test
Summary:
fix the values in the test, the test was broken due to changes in jq

jq used to provide incorrect rounding.

Reviewed By: ahornby

Differential Revision: D31390426

fbshipit-source-id: ab4d7014109d23aa5b4fb95db4f485cea70e5b05
2021-10-06 03:32:56 -07:00
Egor Tkachenko
be8d3b9f5e Extract DerivedGeneration into library
Summary: I'm going to reuse that derived type for unit-test in 2DService. And for that I need to make it library.

Reviewed By: HarveyHunt

Differential Revision: D31340518

fbshipit-source-id: 3960c0d3ae9a72e1fa6dc9afb170c0c708b3cdf8
2021-10-06 03:20:05 -07:00
CodemodService Bot
c2564498eb Daily common/rust/cargo_from_buck/bin/autocargo
Reviewed By: krallin

Differential Revision: D31426792

fbshipit-source-id: 11c5e29f3c4deaa74e7b2f8ab2eec8940cb5fe4a
2021-10-06 02:06:23 -07:00
Xavier Deguillard
11c0481570 store: remove LocalStore usage from BackingStore::getRootTree
Summary:
Writing to the LocalStore is purely the responsability of the
LocalStoreCachedBackingStore and not of the individual BackingStore. Thus, they
cannot assume that the root Tree is actually stored in it and should just
directly import it.

Reviewed By: chadaustin

Differential Revision: D31340206

fbshipit-source-id: 0f485ceb9fa71f7a7bdc8aaefaa850540075c88c
2021-10-05 19:59:27 -07:00
Xavier Deguillard
7ebfd3791f service: speedup glob handler by 26%
Summary:
Looking at Instruments, when issuing tons of glob queries to EdenFS, EdenFS
appears to be spending a very large amount of time adding tasks to the
UnboundedTaskExecutor. Since globs are expected to be fast, we can afford to
execute them inline, reducing this overhead and speeding up glob queries.

Reviewed By: chadaustin

Differential Revision: D31289485

fbshipit-source-id: 428fff9f5fea65073b2a061dc7070d63ae36d95d
2021-10-05 19:10:52 -07:00
Xavier Deguillard
0965da43b3 inodes: eliminate quadratic copies of GlobResult
Summary:
The globbing algorithm is recursive and returns its own glob results merged
with its children's glob results. The merging is done by simply copying the
children's glob result and returning it. What this means is that a single
GlobResult will be copied K times, with K being the recursion depth at which it
was created. This makes the total number of copies be O(K*N) with N the result
length.

Since we can simply avoid these copies by simply creating the GlobResult in a
shared vector, we can avoid the copies entirely at the expense of taking a
lock.

Reviewed By: chadaustin

Differential Revision: D31288036

fbshipit-source-id: ae8a98a01eab2ba7f23908d347d7a4ec199cdfab
2021-10-05 19:10:52 -07:00
Xavier Deguillard
2d6df0c676 inodes: return an ImmediateFuture from TreeInode::getOrLoadChild
Summary:
In the case where a child is already loaded, this allows the code to not
allocate a SemiFuture, which should benefit code that repeatly calls it on
already loaded inodes. One typical example is the globbing entry point that
needs to find the TreeInode where the glob needs to be applied on.

Reviewed By: chadaustin

Differential Revision: D31283398

fbshipit-source-id: 76f82d74f2a45ee2b3b9bf442d47c0a2262bced9
2021-10-05 19:10:51 -07:00
Xavier Deguillard
4d44482085 inodes: switch EdenMount::getInode to ImmediateFuture
Summary:
One layer about getChildRecursive is getInode, let's make this one use an
ImmediateFuture too.

Reviewed By: chadaustin

Differential Revision: D31283397

fbshipit-source-id: 8bc524bea857d6ec5bc045d6e3383d38133c3b38
2021-10-05 19:10:51 -07:00
Yan Soares Couto
0cd8e64c51 Use auto_wire on all api objects where it's possible
Summary:
This diff uses the helper proc macro added on the previous diff to simplify the code for dozens of api objects.

Over 1000 lines deleted :)

Examples of structs that couldn't be migrated:
- Wire structs that didn't rename the fields to numbers. (e.g. WireHistoryRequest) (would need some extra migration)
- enums  (not currently supported by the proc macro)
- Wire structs that didn't map directly to their non-wire counterparts (e.g. WireSnapshotState)

I added some comments with possible future improvements, but didn't pursue them right now as they are significantly less useful than this diff itself, which covers most of the cases.

Reviewed By: ahornby

Differential Revision: D31057140

fbshipit-source-id: 88a867ba2cdfedf6a96a8ca3718508073822b962
2021-10-05 16:32:03 -07:00
Yan Soares Couto
1e461b5028 Proc macro for default wire implementation
Summary:
This diff implements a proc macro that derives a "wire" object from a simple api obj for edenapi. It is enough to cover most of the wire objects and simplify code a lot.

Differences from initial proof of concept:
- Renamed `default_wire` to `auto_wire`. I'm happy to rename again if preferred.
- Fields must have `#[id(X)]` notation.
- Fields with default values are not serialized.
- Arbitrary implementation also is derived

This diff only uses this proc_macro on one Api Object, in order to make reviewing it easier, focusing on the implementation of the macro. The next diff uses this macro in all possible objects, actually doing the bulk of the refactoring.

Reviewed By: ahornby

Differential Revision: D31054734

fbshipit-source-id: d6136faf84492983ca69461fe243e830021b2f66
2021-10-05 16:32:03 -07:00
Zeyi (Rice) Fan
3785bf7ff2 config: add a new knob
Reviewed By: chadaustin

Differential Revision: D31163226

fbshipit-source-id: 96dcba2bd62a0ce879a92a94050873c5725adce4
2021-10-05 15:52:49 -07:00
Muir Manders
5eb5c5e7bb native status: support HGPLAIN color suppression
Summary:
Plus a minor refactoring to use the io::IsTty trait in edenfs_client::status instead of calling into libc directly.

This commit reintroduces 218f06a4e648 after reversion in e9ef7c2142d0. Windows build broke because I accidentally left a stray `#[cfg(unix)]` above unrelated use line.

Differential Revision: D31248594

fbshipit-source-id: ddee62d9dc4d0b99d2e67fe9b757748ab501e030
2021-10-05 11:03:33 -07:00
Jun Wu
d354aa787f edenapi: drop debug config eprint support
Summary:
Setting EDENSCM_LOG can achieve the same effect. So let's remove the redundant
logic. This also avoids overhead constructing the strings when tracing is off.

Reviewed By: yancouto

Differential Revision: D31359046

fbshipit-source-id: db53cc16f1efcf6111535090a3eadec19b888329
2021-10-05 09:47:19 -07:00
Yan Soares Couto
075a4a1148 filenodes: switch to manager-based derivation
Summary: Same as D30974102 (91c4748c5b) but for mercurial filenodes.

Reviewed By: markbt

Differential Revision: D31170597

fbshipit-source-id: fda62e251f9eb0e1b6b4aa950d93560b1ff81f67
2021-10-05 06:26:34 -07:00
Yan Soares Couto
ddba827364 Fix u64/i64 conversion in tests
Summary:
This fixes these tests (which are blocking the deployment):
```
FAILED	eden/mononoke/mononoke_types:mononoke_types-unittest - content_metadata::test::content_metadata_blob_roundtrip
FAILED	eden/mononoke/mononoke_types:mononoke_types-unittest - content_metadata::test::content_metadata_thrift_roundtrip
FAILED	eden/mononoke/mononoke_types:mononoke_types-unittest - file_contents::test::file_contents_blob_roundtrip
FAILED	eden/mononoke/mononoke_types:mononoke_types-unittest - file_contents::test::file_contents_thrift_roundtrip
```

They were broken since the update of quickcheck, as it now checks with big values for ints, which broke these tests as the code didn't do a good job of converting between u64/i64. This diff fixes that.

Usually, we just want to use `as` conversion everywhere. Thrift only stores i64. but it's fine to store a negative value that will correctly overflow back to the correct positive value of u64. In practice though, this shouldn't really happen, as these are sizes, and we're never getting anywhere near the u64 limit.

Using `TryInto` doesn't cause these overflows, but for just storage, it makes things work in less cases.

Reviewed By: krallin

Differential Revision: D31379001

fbshipit-source-id: feeb87a62148f97b3bd467e8c2ef2156c8e3329a
2021-10-05 05:57:45 -07:00
Juan Pablo Lopez
76060c7d5f Provide support for x2pagentd using unix socket
Summary: Establish a http connection to x2pagentd using a unix socket.

Reviewed By: mzr

Differential Revision: D31204332

fbshipit-source-id: d1f98dc0e4eae4fb918cd77a9571e1a2da7d7ed2
2021-10-05 04:01:33 -07:00
Mark Juggurnauth-Thomas
af7014bfe5 mononoke_api: ignore copy-replaced files when computing diff renames
Summary:
If a file is replaced by a move or copy from another file, we should ignore
this copy info when computing the renames in a diff.  If we do not, then we
will fail to include the deletion of the copy source in the case of a move, and
the copy information doesn't add anything anyway as the destination file will
just show as modified.

Reviewed By: mitrandir77

Differential Revision: D31180151

fbshipit-source-id: c89a8ae26a516fd958406bb967a587b3b6c36a48
2021-10-04 07:16:35 -07:00
Mark Juggurnauth-Thomas
053a54b7e8 bounded_traversal: fix warning
Reviewed By: yancouto

Differential Revision: D31339096

fbshipit-source-id: 422c078ae79ac668dd39386eaf64cf848b0a1aa8
2021-10-04 06:59:25 -07:00
Mark Juggurnauth-Thomas
8b21fe64fd git_types: switch to manager-based derivation
Summary: Switch derivation of Git trees to use the `DerivedDataManager`.

Reviewed By: yancouto

Differential Revision: D31303798

fbshipit-source-id: 193f5d373a56a0d1099f49db76758227d15c3762
2021-10-04 06:59:25 -07:00
Mark Juggurnauth-Thomas
9aa5c6a64a dangerous_override: add overrides for derived data manager
Summary: Add overrides for changesets, filenodes and bonsai_hg_mapping in the derived data manager.

Reviewed By: yancouto

Differential Revision: D31378456

fbshipit-source-id: b1faa543ca65fa041d2d0ddc908ea5fb950d023a
2021-10-04 06:59:25 -07:00
Jan Mazur
083a9ad35c Vendor curl-sys crate with patched curl
Summary:
Vendoring this patch: https://github.com/curl/curl/pull/7737 to curl-sys rust crate. On windows the hg client is using curl that's bundled with sys-curl. I need this patch to have unix domain sockets working in hg client on windows.

I had to manually vendor https://raw.githubusercontent.com/mzr/curl/57e7ec4dbe4dd2831de51f2644879387d2ea7b44/docs/INSTALL because reindeer didn't do it. IDK why.

oss-eden-{darwin,linux,windows}-getdeps fail with:
```
FAILED: eden/scm/lib/backingstore/CMakeFiles/rust_backingstore.cargo eden/scm/lib/backingstore/debug/libbackingstore.a eden/scm/lib/backingstore/release/libbackingstore.a
cd /data/sandcastle/temp/fbcode_builder_getdeps/shipit/eden/eden/scm/lib/backingstore && /data/sandcastle/temp/fbcode_builder_getdeps/installed/cmake-hQhVzQT-WzFKTeqXjLxo5lLi8IG4_MjX2-YRqptCUVs/bin/cmake -E remove -f /data/sandcastle/temp/fbcode_builder_getdeps/shipit/eden/eden/scm/lib/backingstore/Cargo.lock && /data/sandcastle/temp/fbcode_builder_getdeps/installed/cmake-hQhVzQT-WzFKTeqXjLxo5lLi8IG4_MjX2-YRqptCUVs/bin/cmake -E env CARGO_TARGET_DIR=/data/sandcastle/temp/fbcode_builder_getdeps/build/eden/eden/scm/lib/backingstore CARGO_HOME=/data/sandcastle/temp/fbcode_builder_getdeps/build/eden/_cargo_home cargo build --release -p backingstore --features fb
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
error: failed to calculate checksum of: /data/sandcastle/boxes/eden-trunk-hg-fbcode-fbsource/third-party/rust/vendor/curl-sys-0.4.45+curl-7.78.0/curl/docs/INSTALL
Caused by:
  failed to open file `/data/sandcastle/boxes/eden-trunk-hg-fbcode-fbsource/third-party/rust/vendor/curl-sys-0.4.45+curl-7.78.0/curl/docs/INSTALL`
Caused by:
  No such file or directory (os error 2)
```

Not idea how to fix this. Seems related to the fact that reindeer didn't vendor docs/INSTALL.

# EDIT:
# It might been caused by some bug in hg. now it's fine
# The failure in oss-eden-linux-getdeps looks unrelated (something with rocksdb)

Reviewed By: krallin

Differential Revision: D31370778

fbshipit-source-id: a1245f8cb6b58f5765e34c95dfd78325a8e6e457
2021-10-04 03:13:50 -07:00
Yan Soares Couto
afaadd87cf Add tests for a few missing wire types
Summary:
There were a few Wire objs that didn't have tests because they didn't implement Arbitrary. I added simple implementations for them.

I did it mostly for the request/response types, as they have the other types inside them.

Reviewed By: markbt

Differential Revision: D31019959

fbshipit-source-id: edf283fd79a40b794c89db79c026f1bcaf834a9f
2021-10-01 14:43:33 -07:00
Yan Soares Couto
1a78607026 Add snapshots for wire types and simplify tests
Summary:
This diff adds snapshot tests for most eden api wire types, while at the same time making the testing code much smaller, including tests for "wire" and "serialize" roundtrips.

## Context:
The previous diff had added an easy way to add snapshot tests. This stack aims to simplify the wire protocol code needed to create/modify an endpoint. A good thing to do before that is to add snapshot tests to all wire types, so that if we change them in a refactor, we're confident they still work exactly the same. This will also be useful when a type is changed in the future.

## How this makes tests easier
- In order to create snapshot tests, we need example objects to test with. Luckily we already use a framework for generating example objects (quickcheck::Arbitrary), so the idea here is to use that to make snapshot tests as automatic as possible.
- At the same time, the "wire" and "serialize" roundtrip tests (which also used Arbitrary), can also be made more automatic.

Now, using a simple helper, `auto_wire_tests!(WireObjectName)`, it is possible to derive all three types of tests automatically. This makes the current code smaller, and safer as we now have the additional safety provided by snapshot tests.

## Observations
- Not all wire types had tests implemented for them (I assume because it was too much work doing so, and might have done that myself in the past), I only moved the ones that already had. I'll do another pass and add remaining objects on a following diff.
- There are a couple actual non-refactor changes. I'll add comments explaining those.
- quickcheck crate is using quite an old version. I tried updating but it snowballed into something much more complicated, so I kept using the old version. We'll need to get to it at some point, though.

Reviewed By: markbt

Differential Revision: D31019233

fbshipit-source-id: 30c4a90848d0a5dcaffb89b9a0cd1cebfe4ace55
2021-10-01 14:43:32 -07:00
Yan Soares Couto
30cc107728 Add snapshot test for 2 wire objects
Summary:
This diff adds a way to do snapshot tests in SC, by simply calling a macro. It uses that as an example on two wire objects to show it works.

Motivation:
- Using snapshot testing in wire object will make it super clear what changes are happening on the wire format, which is useful to prevent breakages. (see quip)
- I'm going with approach #3 from proposal, but both #2 and #3 would need snapshot testing.

How?
- I'm using [insta](https://docs.rs/insta/1.8.0/insta/), a rust crate for snapshot testing. Unfortunately it depends on some cargo-specific stuff that doesn't work with buck, but I was able to get it working without patching the package by bypassing the Cargo dependent stuff.
- Insta also provides some cargo exensions on top of it to compare snapshots, which we can't have, so I made it clear on the test errors (see test plan) how to update the snapshots (via flag).

Future
- On future diffs I'll add a snapshot test for all the wire objects, this just adds to a first few to show it works and set up the plumbing.
- Writing objects manually for snapshot tests for **every** wire object would be a large amount of code, and troublesome when adding new (specially big) wire objects. We already have wire objects implementing `Arbitrary`, so I think it should be possible to use this to generate simple snapshot tests on automatically generated objects. (It will need some extra work to make sure they are created consistently.)

Reviewed By: markbt

Differential Revision: D30958077

fbshipit-source-id: 3d8663e7897e5f6eb4b97c24f47b37ef2f138e5a
2021-10-01 14:43:32 -07:00
Genevieve Helsel
d9afd277b7 disable test_mount_state_during_unmount_with_in_progress_checkout on eden-asic-build-and-test
Summary: TSIA

Reviewed By: xavierd

Differential Revision: D30825961

fbshipit-source-id: e920e4e30289fbdcea4cd6d52e6f4d2dfbd3409d
2021-10-01 14:07:24 -07:00
Andrey Chursin
ae684f3993 explicit Hash20 instead of Hash [proxy hash removal 2/n]
Summary:
This is fairly mechanical diff that finalizes split of Hash into ObjectId and Hash20.

More specifically this diff does two things:
* Replaces `Hash` with `Hash20`
* Removes alias `using Hash = Hash20`

Reviewed By: chadaustin

Differential Revision: D31324202

fbshipit-source-id: 780b6d2a422ddf6d0f3cfc91e3e70ad10ebaa8b4
2021-10-01 12:43:26 -07:00
Mark Juggurnauth-Thomas
0430014575 edenfs-client: specify spawner
Summary:
A spawner type is required for new thrift clients, specify the noop one for now.

This also requires regenerating the generated thrift libraries.

Reviewed By: yancouto

Differential Revision: D31338518

fbshipit-source-id: cbecf3ec6f9678918ca459c19f1cc160214fadfd
2021-10-01 11:35:17 -07:00
Jun Wu
2fd0fc5aaf pullcreatemarkers: disable pullcreatemarkers in pull command
Summary:
We now use post-pull hook to mark commits as landed:

  hooks.post-pull.marklanded=hg debugmarklanded

So there is no need to mark them inside the pull command. In fact, it seems
something is wrong (phases aren't invalidated properly?) so that the
pullcreatemarkers logic might actually hide commits during pull incorrectly:

  commit 1e964a4302c03e5ae48e5b85b0fc0bf27f847b09
  Author: metalog <metalog@example.com>
  Date:   Tue Sep 28 17:29:49 2021 +0000

      pull
      Parent: a4511a83cf862cc7216c15d83a3f4ff9d3b3241b
      Transaction: pullcreatemarkers

      RootId: 18d81a1531ecea65affe83c25804c790cac57c59

  diff --git a/visibleheads b/visibleheads
  index eb58137..da5f45a 100644
  --- a/visibleheads
  +++ b/visibleheads
  @@ -1,16 +1,6 @@
   v1
  -e82de77adcc261cb306dafeb6cbe15f26f7de768
  -91cf8c4b47e433acbe3f774e608eee42a3ad089d
   8c071e5aa26d920f1b88c4b1cd10f6e946d4312a
   536ff436cde4ec53d74c02ae2c5ed6f60609e01a
  -e49b834a6ff9b61a95a743d22703dc6634f2918c
  -c53b65542d4583ae82835144ac7f72dce7a6f335
   1010312ed7ac683c5e97ad765cdbcb4927ddf62c
  -a563a5f35d2df4c54ba1fe2401aa1cd929a218bf
  -55c0c7483c6cd85114a9be587d11c87aaceaeff4
  -74079567b677c5799b6c67855683ec346ebc3cce
   33b5fd6055da2284cf5224f70e1bb9791ed87641
  -ec221dff2393793f7b1e11ea9f9e9ea87b79da1f
  -cc466614d43a6af24520b7a81653435f4a614fbb
   2d7466be885e757d2d41630a3148fd31f5199ffa
  -226c8136603dcbcc408133a2122e93fc045527fa

As we're here, document some other config options existed in the code.

Reviewed By: DurhamG

Differential Revision: D31295709

fbshipit-source-id: e26c728215a209ab5dfaee7a84daece8197a1cc4
2021-10-01 11:13:32 -07:00
Xavier Deguillard
68bbbc8123 inodes: switch TreeInode::getChildRecursive to ImmediateFuture
Summary:
With the lookup processor now returning an ImmediateFuture, we can focus on its
caller, starting with getChildRecursive.

Reviewed By: chadaustin

Differential Revision: D31283396

fbshipit-source-id: 97abc57b9efe3540c5770aa952995c257e6eda4b
2021-10-01 10:43:06 -07:00
Xavier Deguillard
b613cf82f6 inodes: make LookupProcessor use ImmediateFuture
Summary:
While profiling glob queries sent by Buck, I noticed that EdenFS spends almost
as much time resolving the inode to perform the glob on as EdenFS takes to
actually perform the glob. Futures related overhead shows up as predominent,
thus let's convert these to ImmediateFuture to speed this up.

Reviewed By: chadaustin

Differential Revision: D31283395

fbshipit-source-id: 7355ddf7498f722ed8ec2989f010a28fb15c293f
2021-10-01 10:43:05 -07:00
Xavier Deguillard
8c26a8ffe5 immediatefuture: speed up by 40%
Summary:
While std::variant is convenient, they are both slow to compile, and the
compiler cannot optimize it as well as a manually written tagged union. Since
ImmediateFuture is performance critical for EdenFS, let's use a tagged union
and speed them up by an additional 40%.

Reviewed By: chadaustin

Differential Revision: D31272296

fbshipit-source-id: e34be4489a596d3577b3bd900a1f20d6c7d8b693
2021-10-01 10:43:05 -07:00
Xavier Deguillard
5b5157ab51 immediatefuture: split get with and without timeout
Summary:
The max duration would cause UBSAN failures due to folly's SemiFuture code
multiplying the value which understandably cannot be represented. Splitting the
function is easy and avoids the problem entirely.

Reviewed By: genevievehelsel

Differential Revision: D31272297

fbshipit-source-id: c15ca70ad771c11b4f68bb9974422c0986d4928b
2021-10-01 10:43:05 -07:00
Xavier Deguillard
0505b5f1d6 benchmarks: add a search root for glob benchmark
Summary:
When Buck is using the EdenFS globber, the searchRoot argument is set, thus
let's add a new argument to the benchmark to simulate a Buck workflow.

Reviewed By: chadaustin, genevievehelsel

Differential Revision: D31283399

fbshipit-source-id: 5e32b2aceb6090e26e88cf7f0d163448d56107d4
2021-10-01 10:43:05 -07:00
Andrey Chursin
0af2511a3f separate out ObjectId [proxy hash removal 1/n]
Summary:
The goal of this stack is to remove Proxy Hash type, but to achieve that we need first to address some tech debt in Eden codebase.

For the long time EdenFs had single Hash type that was used for many different use cases.

One of major uses for Hash type is identifies internal EdenFs objects such as blobs, trees, and others.

We seem to reach agreement that we need a different type for those identifiers, so we introduce separate ObjectId type in this diff to denote new identifier type and replace _some_ usage of Hash with ObjectId.

We still retain original Hash type for other use cases.

Roughly speaking, this is how this diff separates between Hash and ObjectId:

**ObjectId**:
* Everything that is stored in local store(blobs, trees, commits)

**Hash20**:
* Explicit hashes(Sha1 of the blob)
* Hg identifiers: manifest id and blob hg ig

For now, in this diff ObjectId has exactly same content as Hash, but this will change in the future diffs. Doing this way allows to keep diff size manageable, while migrating to new ObjectId right away would produce insanely large diff that would be both hard to make and review.

There are few more things that needs to be done before we can get to the meat of removing proxy hashes:

1) Replace include Hash.h with ObjectId.h where needed
2) Remove Hash type, explicitly rename rest of Hash usages to Hash20
3) Modify content of ObjectId to support new use cases
4) Modify serialized metadata and possibly other places that assume ObjectId size is fixed and equal to Hash20 size

Reviewed By: chadaustin

Differential Revision: D31316477

fbshipit-source-id: 0d5e4460a461bcaac6b9fd884517e129aeaf4baf
2021-10-01 10:25:46 -07:00
Yan Soares Couto
648dea3cfc Add sampling of location_to_hash endpoint
Summary:
https://pxl.cl/1Qh3j
This is the most called edenapi endpoint by far. If we sample logging of it, we can increase the retention of the scuba table.

if we wish, it's possible to not change retention for some "non-trivial" requests, but I haven't done that.

Reviewed By: liubov-dmitrieva

Differential Revision: D31277391

fbshipit-source-id: ee19e9daa4cd39c5d3eac1063e82aa40fc108bc7
2021-10-01 03:37:51 -07:00
Yan Soares Couto
1d9afdf9fb Use short gotham request id to save storage space
Summary:
This is used to uniquely identify requests in gotham. It's logged to output, on errors, and on Scuba.

Problem: On scuba, this packs very badly, as it is a large string (36 chars), unique for all requests.

Solution: Let's get a prefix of it, it should reduce size used on scuba. Got a prefix of size 5.

This affects both LFS and EdenApi.

Pros:
- Reduces size
- Very easy fix

Cons:
- More chance of conflict. The space of this id is 16^5 = 10^6. There will surely be conflicts, but maybe that's not a huge deal?

Alternative: Using 8 digits, that's about 4bi ids, which will reduce conflicts significantly in exchange for more space.

Why not use an int id (example: u64), or using other characters in id (not only hex): This would reduce the size of data significantly, but has drawbacks:
- For int, would require a big refactoring, as everything assumes the id to be string. Specially since this goes through client-server, might be complicated.
- Not just getting a prefix means more processing on each request, and means we need to recalculate it everytime.
- Size reduction might not be that big, as scuba already packs stuff pretty well.

Reviewed By: krallin

Differential Revision: D31305547

fbshipit-source-id: 23f6b6cb7de5b7a090864db414d4d71cd68c4946
2021-10-01 03:37:51 -07:00
Yan Soares Couto
da13975a4f Fix quickcheck update breakages
Summary: D31115820 (ae87b82eaf) updated quickcheck, but there's some stuff we need to fix forward. This diff fixes the remaining failures I could find.

Reviewed By: farnz

Differential Revision: D31305392

fbshipit-source-id: a6684d47833bc0fd933751c13cdd71392cb1833b
2021-10-01 03:37:51 -07:00
Liubov Dmitrieva
b09728399c use hg cloud upload command if usehttpupload enabled
Summary:
use `hg cloud upload` command if usehttpupload enabled

For few users who opted out of cloud sync, we should use `hg cloud upload` command instead of `hg cloud backup` if usehttpupload enabled

Reviewed By: markbt

Differential Revision: D31205919

fbshipit-source-id: 7619e7b299e19a7626782e7b3c1a69e7cd7dbc1b
2021-10-01 03:14:28 -07:00
Mateusz Kwapich
8ca8816fdd fix the semaphore deadlock
Summary:
Somehow before this fix I've seen us runing out of semaphores and deadlocking
because not freeing semaphores immediately after finishing running the
function requiring git repo.

Reviewed By: mojsarn

Differential Revision: D31310626

fbshipit-source-id: ba12b2d4918ecc30ca0aa6ff011176f7634badf9
2021-10-01 02:33:43 -07:00
Jeremy Fitzhardinge
66e1fc5370 rust: rerun autocargo after thrift/rust changes
Summary: Need this for cargo check/rust-analyzer to work.

Reviewed By: guswynn

Differential Revision: D31319911

fbshipit-source-id: ebd3fa72d8fc3667391a2067f95cab9e5f53301f
2021-09-30 17:40:04 -07:00
Jonathan Keljo
542e84d8fc Enable unbounded_depth feature for serde_json
Summary: I'm parsing some deeply-nested JSON, and it's running into the limit. This feature enables a potential footgun, but even with the feature enabled you have to add code to reach for said footgun.

Reviewed By: jsgf

Differential Revision: D31284743

fbshipit-source-id: 00ea5d7d7db8bdeb878d48fe390831f39e007409
2021-09-30 14:56:43 -07:00
Jun Wu
9d3faf8af5 smartset: avoid using outdated rev<->node mapping or nameset
Summary:
When calling `changelog.idmap`, it takes a snapshot of the IdMap.

When flushing the changelog, the IdMap might change. Some Ids might get
re-assigned.

For code (phases?) that keeps the `nameset` for a while the idmap might become
out-dated after a changelog flush. That might be a cause of removing visible
heads incorrectly.

Let's change `_torev` and `_tonode` to properties so they always take the
"latest" idmap snapshot.

Reviewed By: DurhamG

Differential Revision: D31296057

fbshipit-source-id: 1b3fec5d21649eab772ab3a150a3182a18b94edf
2021-09-30 10:28:35 -07:00
Jeremy Fitzhardinge
f5451a34bc scs: turn all comments into doc comments
Summary:
Use `///` for comments so they're converted
into doc comments so that they'll be rendered in the Rust docs (eg
https://www.internalfb.com/intern/rustdoc/eden/mononoke/scs/if:source_control-rust/source_control/client/trait.SourceControlService.html,
or hover docs in vs code).  It's already pretty well documented, so it
would be nice to take advantage of it.

Reviewed By: krallin

Differential Revision: D31288804

fbshipit-source-id: 9f2b3e0966832791c6ce39574cafdc71f8c032dd
2021-09-30 09:42:06 -07:00
Stanislau Hlebik
7221a52147 mononoke: add new idx flag
Summary:
It was added on the client side in D30686450 (7eb11cb392) to handle octopus merges correctly,
let's add it on mononoke as well, otherwise new_streaming_clone fails to parse
a revlog.

Reviewed By: mitrandir77

Differential Revision: D31305651

fbshipit-source-id: 976d7fdb8775f859e4732fd8a68f9b28f04ce4f9
2021-09-30 07:01:03 -07:00
Jan Mazur
21bb613c25 unix domain socket support for httpclient.HTTPConnection
Summary: This implements unix socket support for mercurial's HTTPConnection so commitcloud can use it.

Reviewed By: ahornby

Differential Revision: D31229256

fbshipit-source-id: a610c3c34be608ac2d9b41f3a7b6b62b44227b94
2021-09-30 05:56:44 -07:00
Qinfan Wu
dbac51105d Fix broken builds caused by rand upgrade
Summary:
A previous commit updated `rand` from 0.7 to 0.8. One breaking change introduced was that `Alphanumeric` now samples `u8` instead of `char`.

See https://docs.rs/rand/0.7.3/rand/distributions/struct.Alphanumeric.html and https://docs.rs/rand/0.8.4/rand/distributions/struct.Alphanumeric.html.

Reviewed By: bolinfest

Differential Revision: D31298553

fbshipit-source-id: 5d0d588550f17bac5ca4788748ec3f873398bf35
2021-09-29 23:09:58 -07:00
Egor Tkachenko
0bfc399700 Map requests to shared future for derivation.
Summary:
We want to reduce duplicated work. Since requests will be consistently hashed, each instance of service will receive some set of requests from multiple clients. By storing requests together with shared future derivation, any client can get the state of derivation.
In addition upon receiving requests we can clean up the map to remove already completed futures so the map will not grow indefinitely.

Reviewed By: StanislavGlebik

Differential Revision: D30776322

fbshipit-source-id: 961055f8b3328378451edd677506d7e716a9afd2
2021-09-29 16:10:04 -07:00
Muir Manders
e50031478a runlog: include progress information
Summary:
Now the runlog for an hg command invocation includes any progress bar metadata. I repurposed the existing rust progress thread to also upate the runlog progress (only if it has changed).

To avoid race conditions with the main thread writing the final "exit" runlog entry, updating the runlog progress is a no-op if the runlog's exit code has already been set.

Reviewed By: quark-zju

Differential Revision: D31065260

fbshipit-source-id: 181661cb06ab2910d8a0e41f5aa767528eb234f5
2021-09-29 16:05:37 -07:00