Commit Graph

62320 Commits

Author SHA1 Message Date
Alex Hornby
92ce1e74d2 mononoke: allow passing of repo name through walker
Summary: preparation for multi repo, get the repo name into ErrorKind::NotTraversable

Reviewed By: StanislavGlebik

Differential Revision: D25541444

fbshipit-source-id: 8fd99d5d3f144d8a3a72c7c33205ae58bd5f1ae2
2020-12-23 02:08:22 -08:00
Alex Hornby
c71c9c1459 mononoke: factor out per repo params in walker
Summary:
In preparation for having the walker able to scrub multiple repos at once, define parameter structs.  This also simplifies the code in tail.rs.

The param objects are:

* RepoSubcommandParams - per repo params that can be setup in setup_common and are consumed in the subcommand.  They don't get passed through to the walk

* RepoWalkParams - per repo params that can be setup in setup_common and will get passed all the way into the walk.rs methods

* JobWalkParams - per job params that at can be setup in setup_common and will get passed all the way into the walk.rs methods

* TypeWalkParams - per repo params that need to be setup in the subcommands, and are passed all the way into walk.rs

Reviewed By: StanislavGlebik

Differential Revision: D25524256

fbshipit-source-id: bfc8e087e386b6ed45121908b48b6535f65debd3
2020-12-23 02:08:22 -08:00
Alex Hornby
a36ca7efa8 mononoke: factor out arg parsing functions in walker
Summary: parsing of progress options an sampling options was same in each subcommand,  move them to functions in setup.rs

Reviewed By: StanislavGlebik

Differential Revision: D25524255

fbshipit-source-id: a2f48814f24aa9b3a158cb7d4abbfc2c0c338305
2020-12-23 02:08:22 -08:00
Alex Hornby
45d9b20949 mononoke: simplify open_blobrepo_given_datasources parameters
Summary: Simplify open_blobrepo_given_datasources  parameters to pass less arguments, make it so can pass the sql_factory by reference.

Reviewed By: krallin

Differential Revision: D25524254

fbshipit-source-id: c324127f42c53a52f388d303e310014f4fa0d7bb
2020-12-23 02:08:22 -08:00
Alex Hornby
ef501ba297 mononoke: update walker setup_common to async fn
Summary: Shifts things left a bit

Reviewed By: krallin

Differential Revision: D25524257

fbshipit-source-id: ab0979b7e5370c1ad142ecabc1d27fea549a3342
2020-12-23 02:08:22 -08:00
Alex Hornby
9985458fa1 mononoke: prepare walker blobstore for multiple repo jobs
Summary: Allows the walker blobstore code to be used by more than one blobrepo.  This is a step to reduce the number of jobs needed to scrub small repos.

Reviewed By: StanislavGlebik

Differential Revision: D25422937

fbshipit-source-id: e2d11239f172f50680bb6e10dd60026c9e6c3c3d
2020-12-23 02:08:22 -08:00
Alex Hornby
a7658c112e mononoke: step to hg changesets via bonsai in walker
Summary:
By doing the hg to hg steps via bonsai I will later introduce a check if the bonsai is in the current chunk of commits to be processed as part of allowing walker checkpoint and restart.

On its own this is a minor change to the number of nodes the walk will cover as seen in the updated tests.

Reviewed By: krallin

Differential Revision: D25394085

fbshipit-source-id: 3e50cf76c7032635ce9e6a7375228979b2e9c930
2020-12-23 02:08:21 -08:00
Alex Hornby
422774f46d mononoke: track if hg is derived in walker bonsai steps
Summary: This is in preparation for all walker hg to hg steps (e.g HgChangeset to Parent HgChangeset) going via Bonsai, which without this would continually check if the filenodes are derived

Reviewed By: krallin

Differential Revision: D25394086

fbshipit-source-id: bb75e7ddf5b09f9d13a0f436627f4c3c95e24430
2020-12-23 02:08:21 -08:00
svcscm
2f98f6233b Updating submodules
Summary:
GitHub commits:

e30dc6b7f9
1de27792ea

Reviewed By: wittgenst

fbshipit-source-id: 6379380d5427370cc2bfae747873739fdc9318fc
2020-12-23 02:08:21 -08:00
svcscm
5b3e033158 Updating submodules
Summary:
GitHub commits:

b0d5160139
3909c78c54
06ab466b50
69bbfc4b7d
32bef3f12c
f4db3e4119
ca016562f5
63c17e1186
6309b1a525
a84fe7bd3c
580b4e5a6c
e9e695adec
a8a3bb24bf

Reviewed By: wittgenst

fbshipit-source-id: 2f8e317c8c48c3681b6805f254783f2aeaf847dc
2020-12-22 17:42:27 -08:00
Jun Wu
3b6bec975e dag: impl PartialOrd for VerLink
Summary:
`PartialOrd` was suggested by sfilipco. Note `Option<std::cmp::Ordering>` is
similar to `Side` in terms of expressiveness. `PartialOrd` can be written
using shorter symbols (`<=`, etc) so it's easier to understand.

The `compatible` family APIs were replaced by `partial_cmp` APIs.

There are some minor differences:
- Bitwise or used by union set is no longer supported. `Hints::union` was
  added as a replacement.
- `Option<T>` implements full order. `Some(T) > None`. This is different
  from `compatible_dag` and `compatible_id_map` APIs. Additional `> None`
  checks were added for correctness.

Reviewed By: sfilipco

Differential Revision: D25652784

fbshipit-source-id: 51d88948fa556300678050088c06e9dda09cbf98
2020-12-22 10:37:01 -08:00
Jun Wu
c656a5da77 configparser: remove a compiler warning
Summary:
```
warning: variable does not need to be mutable
   --> eden/scm/lib/configparser/src/config.rs:448:21
    |
448 |                 let mut values_copy = values.clone();
    |                     ----^^^^^^^^^^^
    |                     |
    |                     help: remove this `mut`
    |
    = note: `#[warn(unused_mut)]` on by default
```

Reviewed By: sfilipco

Differential Revision: D25625453

fbshipit-source-id: 8475056a87095f9ba633282666e6d3fee864074b
2020-12-22 10:26:39 -08:00
svcscm
bbc7461afb Updating submodules
Summary:
GitHub commits:

86e67fcf89
e124ab8e2c
99eb6d35af

Reviewed By: wittgenst

fbshipit-source-id: dd669b3c5afbeac87c5fb4a6ac11095be794351f
2020-12-22 10:26:39 -08:00
svcscm
f9e8959b32 Updating submodules
Summary:
GitHub commits:

bbb76de635
ab1dd4470d
2c49f1648b
e45e8964e6
046a69f909
861b0d1a49
ffc985a473
30f71dbe14
5c093ef720
78a9049483
26f7cfc486
0e6ac9d259
c3600d45a8

Reviewed By: wittgenst

fbshipit-source-id: 47d97a3354d690abd42e8e8e0e755eb896a66070
2020-12-21 15:46:40 -08:00
Arthur Kushka
60affd571a Forced watchman daemon to always operate in non elevated mode on Windows (#878)
Summary:
Recently I found that its impossible to access elevated Watchman daemon from non-elevated process on Windows.

```
events.js:174
      throw er; // Unhandled 'error' event
      ^
Error: connect EPERM \\.\pipe\watchman
    at PipeConnectWrap.afterConnect [as oncomplete] (net.js:1106:14)
Emitted 'error' event at:
    at Socket.<anonymous> (C:\open\ovrsource\unity\socialvr\_js\node_modules\fb-watchman\index.js:118:12)
    at Socket.emit (events.js:198:13)
    at emitErrorNT (internal/streams/destroy.js:91:8)
    at emitErrorAndCloseNT (internal/streams/destroy.js:59:3)
    at process._tickCallback (internal/process/next_tick.js:63:19)
```

To fix this, it was suggested by wez to use [his library](https://github.com/wez/EleDo) to force Watchman daemon always start in normal mode on Windows. In this stack of commits I did integrated his library into project and used it to force daemon restart in normal mode when spawned from elevated terminal.

To make it happen, I checked-in library sources and created proxy project which depends on the initial library and contains header bindings and cmake configuration. I did copy pasted Rust cmake macroses from another facebook project - eden, and also created analogue of autogen.sh but for Windows - autogen.cmd.

Pull Request resolved: https://github.com/facebook/watchman/pull/878

Test Plan:
Launch elevated terminal
Start watchman.exe produced from sources
Observe daemon starting and answering
In process monitor, observe watchman.exe process running under user group

Reviewed By: fanzeyi

Differential Revision: D25595879

Pulled By: arhelmus

fbshipit-source-id: 15eb29adcf5bd4a5708b6533a1b2bacbf13f431c
2020-12-21 15:17:27 -08:00
svcscm
8884d39bd0 Updating submodules
Summary:
GitHub commits:

376ad8c999
e5f8e90e46
4542861223

Reviewed By: wittgenst

fbshipit-source-id: fc262853095685948ff7449ca4389d960c3836f3
2020-12-21 15:17:27 -08:00
svcscm
d3cbfce736 Updating submodules
Summary:
GitHub commits:

9cc99a5cac
0f1a42ada3

Reviewed By: 2d2d2d2d2d

fbshipit-source-id: 140263f07feb5ea1211425e2f7b646cbc9be5740
2020-12-21 10:24:00 -08:00
svcscm
2366004b6a Updating submodules
Summary:
GitHub commits:

9cc45ef416

Reviewed By: 2d2d2d2d2d

fbshipit-source-id: e1fa9619e9b1fdca84a74fe103d561f907347630
2020-12-20 11:23:48 -08:00
svcscm
f6a9cdc32c Updating submodules
Summary:
GitHub commits:

dc94d317a5

Reviewed By: 2d2d2d2d2d

fbshipit-source-id: 220307efc25f980d369ab5123c9b65b4912c0f1c
2020-12-19 10:51:34 -08:00
svcscm
075cef409d Updating submodules
Summary:
GitHub commits:

59fe375d89

Reviewed By: 2d2d2d2d2d

fbshipit-source-id: 5de20f2d2ce03cb4a4c50b56b55e5ffaaad6fed6
2020-12-19 10:51:34 -08:00
svcscm
03aa468d38 Updating submodules
Summary:
GitHub commits:

b6b477a0c8
12294e46e3
b3b979b93b
a9a4fa5630

Reviewed By: 2d2d2d2d2d

fbshipit-source-id: d0ec02dc4c69a7d67c7a6699daf983ccab697d2a
2020-12-18 17:22:43 -08:00
Pavel Aslanov
d3b5cd5537 convert admin to new type futures
Summary: convert admin tool to new type futures

Reviewed By: markbt

Differential Revision: D25616784

fbshipit-source-id: 3a96a62df109ce6a42437da1393c6c3644392aff
2020-12-18 17:14:23 -08:00
Jun Wu
6132962d30 dag: enable dev-logger
Summary: This makes `RUST_LOG` effective during `cargo test`.

Reviewed By: sfilipco

Differential Revision: D25622972

fbshipit-source-id: b1b28e8d3bb397055815b4ac7cda8fbfac7ea0eb
2020-12-18 16:56:44 -08:00
Jun Wu
48d7297887 dag: remove unnecessary snapshots
Summary:
Some code paths use (expensive) snapshot to be compatible with `Arc::ptr_eq`
compatibility check. With `VerLink` it's more efficient to use `VerLink`
directly. This is potentially more efficient for `VerLink` too because the
`Arc` won't be cloned unnecessarily and `VerLink::bump()` is more likely to
use its optimized path.

Reviewed By: sfilipco

Differential Revision: D25608200

fbshipit-source-id: 1b3ecc5d7ec5d495bdda22d66025bb812f3d68a0
2020-12-18 16:56:44 -08:00
Jun Wu
d0d149d868 dag: use VerLink to track IdMap change compatibility
Summary:
Similar to the previous change. `VerLink` tracks compatibility more accurately.
- No false positives comparing to the current `map_id` approach.
- Less false negatives comparing to the previous `Arc::ptr_eq` approach.

The `map_id` is kept for debugging purpose.

Reviewed By: sfilipco

Differential Revision: D25607513

fbshipit-source-id: 7d7c7e3d49f707a584142aaaf0a98cfd3a9b5fe8
2020-12-18 16:56:43 -08:00
Jun Wu
21a1f61285 dag: invalidate snapshot is no longer required or correctness
Summary:
Previously, snapshots need to be invalidated manually. That is error-prone.
For example, `import_clone_data` forgot to call `invalidate_snapshot`.

With `VerLink`, it's easy to check if snapshot is up-to-date. So let's just
use that and remove the need of invalidating manually.

`invalidate_snapshot` is still useful to drop `version` in `snapshot` so
`VerLink::bump` might be more efficient. Forgetting about it no longer affects
correctness.

Reviewed By: sfilipco

Differential Revision: D25607514

fbshipit-source-id: 5efb489cda1d4875bcd274c5a197948f67101dc1
2020-12-18 16:56:43 -08:00
Jun Wu
9ba0b046c0 dag: use VerLink to track dag change compatibility
Summary:
`VerLink` tracks compatibility more accurately.
- No false positives comparing to the current `dag_id` approach.
- Less false negatives comparing to the previous `Arc::ptr_eq` approach.

The `dag_id` is kept for debugging purpose.

Note: By the current implementation, `dag.flush()` will make `dag`
incompatible from its previous state. This is somewhat expected, as
`flush` might pick up any changes on the filesystem, reassign non-master. Those
can be actually incompatible. This might be improved in the future to detect
reload changes by using some extra information.

Reviewed By: sfilipco

Differential Revision: D25607511

fbshipit-source-id: 3cfc97610504813a3e5bb32ec19a90495551fd3a
2020-12-18 16:56:43 -08:00
Jun Wu
edaed5d4b4 dag: add VerLink to track change compatibility more precisely
Summary:
There are 2 kinds of changes:
- Append-only changes. It is backwards-compatible.
- Non-append-only changes. It is not backwards-compatible.

Previously,
- `Arc::ptr_eq` on snapshot is too fragile. It treats append-only compatible
  changes as incompatible.
  - Even worse, because of wrapper types (ex. `Arc::new(Arc::new(dag))` is
    different from `dag`), even a same underlying struct can be treated as
    incompatible.
- `(map|dag)_id` is too rough. It treats incompatible non-append-only changes
  as compatible.

Add `VerLink` to track those 2 different kinds of changes. It basically keeps a
(cheap) tree so backwards compatible changes will be detected precisely.
`VerLink` will replace IdMap and Dag compatibility checks.

Reviewed By: sfilipco

Differential Revision: D25607512

fbshipit-source-id: 478f81deee4d2494b56491ec4a851154ab7ae52d
2020-12-18 16:56:43 -08:00
Jun Wu
fab3b21289 pydag: expose Dag and IdMap ID via hints
Summary: This makes it easier to investigate fast path issues.

Reviewed By: sfilipco

Differential Revision: D25598077

fbshipit-source-id: 27b7042fb9510321c25371f8c5d134e248b3d5d5
2020-12-18 16:56:43 -08:00
Jun Wu
f626f09bfd dag: add some debug logs about set operations
Summary:
This makes it easier to check if set operations are using fast paths or not by
setting `RUST_LOG=dag=debug`.

Reviewed By: sfilipco

Differential Revision: D25598075

fbshipit-source-id: 1503a195268c0989d5166596f2c8a66e15201372
2020-12-18 16:56:43 -08:00
Jun Wu
eea00a2cb1 dag: add an API for DagAlgorithm identity
Summary:
See the previous diff for context. The new API will be used to check if two
dags are compatible.

Note: It can cause false positive on compatibility checks, which need a
more complex solution. See D25607513 in this stack.

Reviewed By: sfilipco

Differential Revision: D25598079

fbshipit-source-id: f5fc9c03d73b42fadb931038fe2e078881be955f
2020-12-18 16:56:42 -08:00
svcscm
5629071b32 Updating submodules
Summary:
GitHub commits:

8c3b5b2b90
c10fa24424
d7c40bc106
d28d914898
62afa968c2
8246bf2ce1
5adba085bb
f6e5cbceda
6ec1f20b9a

Reviewed By: 2d2d2d2d2d

fbshipit-source-id: db862853b2d4525ad75563f475ce90e359d3a8b7
2020-12-18 16:54:15 -08:00
Jun Wu
3fc8f38541 changelog2: add "lazytext" backend
Summary: The backend is designed to be used by the "debugsegmentclone" command, which does not write revlog.

Reviewed By: sfilipco

Differential Revision: D25624786

fbshipit-source-id: e145128c7b41d78fed495f8da540169f741b674d
2020-12-18 16:47:11 -08:00
Jun Wu
207f755dc0 hgcommits: make revlog optional for the hybrid backend
Summary: This makes it possible to add new commits in a repo without revlog.

Reviewed By: sfilipco

Differential Revision: D25602527

fbshipit-source-id: 56c27a5f00307bcf35efa4517c7664a865c47a43
2020-12-18 16:47:11 -08:00
svcscm
5b5392f1b7 Updating submodules
Summary:
GitHub commits:

17b64c83af
c46887708b
9dc9e2944f

Reviewed By: 2d2d2d2d2d

fbshipit-source-id: 8d1a8859ad87aa3cff3686f8e4d2a322fa641478
2020-12-18 16:47:11 -08:00
svcscm
b25a2c7ba2 Updating submodules
Summary:
GitHub commits:

5386fd4a85
bbc4ede54b
d6693c88a0

Reviewed By: 2d2d2d2d2d

fbshipit-source-id: 62bc4446e39094510fcf386a7cee677c5da3a5f0
2020-12-18 12:45:00 -08:00
Xavier Deguillard
ee48ba3dd6 store: silence howtoeven lint
Summary:
HowToEven believes that both path and manifestNode might be used after being
moved and thus complains about it as that's often what is intended. However,
in C++17, this lint is spurious as both of these variables will be moved after
being copied properly in the first lambda. To silence the linter, let's just
split the combinator chain in 2.

Reviewed By: genevievehelsel

Differential Revision: D25627413

fbshipit-source-id: 1a93ca039310dfd04a3f11bd9c7de32e93057517
2020-12-18 12:10:05 -08:00
svcscm
c0e4a5ce69 Updating submodules
Summary:
GitHub commits:

5f2af60ce4
cf9aab0e91
dfd307bbcf

Reviewed By: 2d2d2d2d2d

fbshipit-source-id: 1bc39fa00a4667454abc0ef635a7fa99150c36f4
2020-12-18 11:38:46 -08:00
Aida Getoeva
7867ce6965 mononoke: fix mysql options
Summary: Because mysql connection pool options had both `conflicts_with(myrouter)` and default values, the binary always failed if myrouter option was provided.

Differential Revision: D25639679

fbshipit-source-id: 21ebf483d4ee88a05db519a14b7e2561b3089ad1
2020-12-18 11:21:38 -08:00
Jun Wu
78005ae5a5 py3: fix test-run-tests.py
Summary:
When running `python3 run-tests.py test-run-tests.py`, some bytes were printed
with `b` prefix.  Convert them to `str`.

Reviewed By: DurhamG

Differential Revision: D25642164

fbshipit-source-id: f1103b24ad88d0d024f6be546bf632141f06ebd1
2020-12-18 10:38:19 -08:00
svcscm
69624fb27c Updating submodules
Summary:
GitHub commits:

20f47e6c4c
1003cdadb9

Reviewed By: 2d2d2d2d2d

fbshipit-source-id: 4a746cd31c6456a062338b2edcc21533f36a6174
2020-12-18 09:55:13 -08:00
Stanislau Hlebik
63690b7aa7 mononoke: remove deprecated method from cross repo sync library
Summary:
A bit of history first. For some time we had a problem in our cross repo sync
library where it used the "current" commit sync version, where "current" meant
"the latest commit sync config version that was added". That was incorrect, and
we migrated away from this model, however there were still a few places that
used get_current_mover_DEPRECATED() mover.

Removing this method from a test file is easy, but it's trickier for
sync_diamond_merge tool. This tool is used to sync a diamond merge from a small
repo to a large repo (this is non-trivial to do, so we don't do it
automatically). To make things simpler this diff requires all involved commits
(i.e. both parents, where to rebase (onto) and root of the diamond merge) to
have the same commit sync config versions.

Reviewed By: markbt

Differential Revision: D25612492

fbshipit-source-id: 6483eed9698551920fb1cf240218db7b7e78f7bd
2020-12-18 09:35:31 -08:00
svcscm
f4a5855547 Updating submodules
Summary:
GitHub commits:

4f641ac16d
4488c1ca63

Reviewed By: 2d2d2d2d2d

fbshipit-source-id: bff8e52fecb027928025eedf4a0560190054f5a0
2020-12-18 08:36:47 -08:00
Liubov Dmitrieva
18b8c66439 introduce some delay befor writing warning
Summary:
The warning will go to debug level logs if the delay is not reached.

The messages about the locks make profoundly bad effect on attitude to commit cloud even if the delay is just 1 second (that is a reasonable delay).

Reviewed By: quark-zju

Differential Revision: D25587459

fbshipit-source-id: 9a09484d590ba04d17a881e0c9c5d543686b934f
2020-12-18 06:32:00 -08:00
Lukas Piatkowski
67b05b6c24 mononoke/integration: make integration tests work under @mode/dev-rust-oss
Summary: Last step in covering Mononoke with mode/dev-rust-oss buck build mode.

Reviewed By: markbt

Differential Revision: D25461223

fbshipit-source-id: 3fa0fa05e8c96476e069990e8f5cc6d56acf38c0
2020-12-18 06:13:32 -08:00
svcscm
cf98c5e0a9 Updating submodules
Summary:
GitHub commits:

2a41f3f09d
d4b2c1ca96

Reviewed By: 2d2d2d2d2d

fbshipit-source-id: 464f0ab42dfb378027f20f60e37e8dc445e5d9c5
2020-12-18 06:13:32 -08:00
svcscm
96359ff3d4 Updating submodules
Summary:
GitHub commits:

087554f8f9
2395a8da5e
1794d85fed
d1802684f9
d545b5f9b0
9dbc250105
085d96b6c4
98d2c092f7

Reviewed By: 2d2d2d2d2d

fbshipit-source-id: 5ac5a6a3fcabc20a2a744131a6177739cdee7669
2020-12-17 16:47:28 -08:00
Aida Getoeva
8b93f52b71 mononoke/mysql: use single static shared connection pool
Summary:
The correct workflow for using a multi-threaded connection pool for multiple DBs is to have a single shared pool for all the use-cases. The pool is smart enough to maintain separate "pools" for each DB locator and limit them to maximum 100 conn per key.

In this diff I create a `OnceCell` connection pool that is initialized once and reused for every attempt to connect to the DB.
The pool is stored in `MononokeAppData` in order to bind its lifetime to the lifetime of Mononoke app. Then it is passed down as a part of `MysqlOptions`.  Unfortunately this makes `MysqlOptions` not copyable, so the diff also contains lots of "clones".

Reviewed By: ahornby

Differential Revision: D25055819

fbshipit-source-id: 21f7d4a89e657fc9f91bf22c56c6a7172fb76ee8
2020-12-17 15:46:30 -08:00
Aida Getoeva
e9f3284b5b mononoke/mysql: make mysql options not copyable
Summary:
In the next diff I'm going to add Mysql connection object to `MysqlOptions` in order to pass it down from `MononokeAppData` to the code that works with sql.
This change will make MysqlOptions un-copyable.

This diff fixed all issues produced by the change.

Reviewed By: ahornby

Differential Revision: D25590772

fbshipit-source-id: 440ae5cba3d49ee6ccd2ff39a93829bcd14bb3f1
2020-12-17 15:46:30 -08:00
Aida Getoeva
8ecb9977a5 mononoke: use parse mysql options in benchmark filestore
Summary:
benchmark_filestore XDB subcommands uses mysql and has option of using either myrouter or mysql. In this diff I used `args::parse_mysql_options` function to parse the arguments instead of manual processing and get a `MysqlOptions` object.
This is needed later to pass a connection pool object through the `MysqlOptions` struct (see the next diff).

Reviewed By: ahornby

Differential Revision: D25587898

fbshipit-source-id: 66fcfd98ad8f3f9e285ca9635d8f625aa680d7ff
2020-12-17 15:46:29 -08:00