Summary:
```
warning: variable does not need to be mutable
--> eden/scm/lib/configparser/src/config.rs:448:21
|
448 | let mut values_copy = values.clone();
| ----^^^^^^^^^^^
| |
| help: remove this `mut`
|
= note: `#[warn(unused_mut)]` on by default
```
Reviewed By: sfilipco
Differential Revision: D25625453
fbshipit-source-id: 8475056a87095f9ba633282666e6d3fee864074b
Summary:
Some code paths use (expensive) snapshot to be compatible with `Arc::ptr_eq`
compatibility check. With `VerLink` it's more efficient to use `VerLink`
directly. This is potentially more efficient for `VerLink` too because the
`Arc` won't be cloned unnecessarily and `VerLink::bump()` is more likely to
use its optimized path.
Reviewed By: sfilipco
Differential Revision: D25608200
fbshipit-source-id: 1b3ecc5d7ec5d495bdda22d66025bb812f3d68a0
Summary:
Similar to the previous change. `VerLink` tracks compatibility more accurately.
- No false positives comparing to the current `map_id` approach.
- Less false negatives comparing to the previous `Arc::ptr_eq` approach.
The `map_id` is kept for debugging purpose.
Reviewed By: sfilipco
Differential Revision: D25607513
fbshipit-source-id: 7d7c7e3d49f707a584142aaaf0a98cfd3a9b5fe8
Summary:
Previously, snapshots need to be invalidated manually. That is error-prone.
For example, `import_clone_data` forgot to call `invalidate_snapshot`.
With `VerLink`, it's easy to check if snapshot is up-to-date. So let's just
use that and remove the need of invalidating manually.
`invalidate_snapshot` is still useful to drop `version` in `snapshot` so
`VerLink::bump` might be more efficient. Forgetting about it no longer affects
correctness.
Reviewed By: sfilipco
Differential Revision: D25607514
fbshipit-source-id: 5efb489cda1d4875bcd274c5a197948f67101dc1
Summary:
`VerLink` tracks compatibility more accurately.
- No false positives comparing to the current `dag_id` approach.
- Less false negatives comparing to the previous `Arc::ptr_eq` approach.
The `dag_id` is kept for debugging purpose.
Note: By the current implementation, `dag.flush()` will make `dag`
incompatible from its previous state. This is somewhat expected, as
`flush` might pick up any changes on the filesystem, reassign non-master. Those
can be actually incompatible. This might be improved in the future to detect
reload changes by using some extra information.
Reviewed By: sfilipco
Differential Revision: D25607511
fbshipit-source-id: 3cfc97610504813a3e5bb32ec19a90495551fd3a
Summary:
There are 2 kinds of changes:
- Append-only changes. It is backwards-compatible.
- Non-append-only changes. It is not backwards-compatible.
Previously,
- `Arc::ptr_eq` on snapshot is too fragile. It treats append-only compatible
changes as incompatible.
- Even worse, because of wrapper types (ex. `Arc::new(Arc::new(dag))` is
different from `dag`), even a same underlying struct can be treated as
incompatible.
- `(map|dag)_id` is too rough. It treats incompatible non-append-only changes
as compatible.
Add `VerLink` to track those 2 different kinds of changes. It basically keeps a
(cheap) tree so backwards compatible changes will be detected precisely.
`VerLink` will replace IdMap and Dag compatibility checks.
Reviewed By: sfilipco
Differential Revision: D25607512
fbshipit-source-id: 478f81deee4d2494b56491ec4a851154ab7ae52d
Summary: This makes it easier to investigate fast path issues.
Reviewed By: sfilipco
Differential Revision: D25598077
fbshipit-source-id: 27b7042fb9510321c25371f8c5d134e248b3d5d5
Summary:
This makes it easier to check if set operations are using fast paths or not by
setting `RUST_LOG=dag=debug`.
Reviewed By: sfilipco
Differential Revision: D25598075
fbshipit-source-id: 1503a195268c0989d5166596f2c8a66e15201372
Summary:
See the previous diff for context. The new API will be used to check if two
dags are compatible.
Note: It can cause false positive on compatibility checks, which need a
more complex solution. See D25607513 in this stack.
Reviewed By: sfilipco
Differential Revision: D25598079
fbshipit-source-id: f5fc9c03d73b42fadb931038fe2e078881be955f
Summary: The backend is designed to be used by the "debugsegmentclone" command, which does not write revlog.
Reviewed By: sfilipco
Differential Revision: D25624786
fbshipit-source-id: e145128c7b41d78fed495f8da540169f741b674d
Summary: This makes it possible to add new commits in a repo without revlog.
Reviewed By: sfilipco
Differential Revision: D25602527
fbshipit-source-id: 56c27a5f00307bcf35efa4517c7664a865c47a43
Summary:
HowToEven believes that both path and manifestNode might be used after being
moved and thus complains about it as that's often what is intended. However,
in C++17, this lint is spurious as both of these variables will be moved after
being copied properly in the first lambda. To silence the linter, let's just
split the combinator chain in 2.
Reviewed By: genevievehelsel
Differential Revision: D25627413
fbshipit-source-id: 1a93ca039310dfd04a3f11bd9c7de32e93057517
Summary: Because mysql connection pool options had both `conflicts_with(myrouter)` and default values, the binary always failed if myrouter option was provided.
Differential Revision: D25639679
fbshipit-source-id: 21ebf483d4ee88a05db519a14b7e2561b3089ad1
Summary:
When running `python3 run-tests.py test-run-tests.py`, some bytes were printed
with `b` prefix. Convert them to `str`.
Reviewed By: DurhamG
Differential Revision: D25642164
fbshipit-source-id: f1103b24ad88d0d024f6be546bf632141f06ebd1
Summary:
A bit of history first. For some time we had a problem in our cross repo sync
library where it used the "current" commit sync version, where "current" meant
"the latest commit sync config version that was added". That was incorrect, and
we migrated away from this model, however there were still a few places that
used get_current_mover_DEPRECATED() mover.
Removing this method from a test file is easy, but it's trickier for
sync_diamond_merge tool. This tool is used to sync a diamond merge from a small
repo to a large repo (this is non-trivial to do, so we don't do it
automatically). To make things simpler this diff requires all involved commits
(i.e. both parents, where to rebase (onto) and root of the diamond merge) to
have the same commit sync config versions.
Reviewed By: markbt
Differential Revision: D25612492
fbshipit-source-id: 6483eed9698551920fb1cf240218db7b7e78f7bd
Summary:
The warning will go to debug level logs if the delay is not reached.
The messages about the locks make profoundly bad effect on attitude to commit cloud even if the delay is just 1 second (that is a reasonable delay).
Reviewed By: quark-zju
Differential Revision: D25587459
fbshipit-source-id: 9a09484d590ba04d17a881e0c9c5d543686b934f
Summary:
The correct workflow for using a multi-threaded connection pool for multiple DBs is to have a single shared pool for all the use-cases. The pool is smart enough to maintain separate "pools" for each DB locator and limit them to maximum 100 conn per key.
In this diff I create a `OnceCell` connection pool that is initialized once and reused for every attempt to connect to the DB.
The pool is stored in `MononokeAppData` in order to bind its lifetime to the lifetime of Mononoke app. Then it is passed down as a part of `MysqlOptions`. Unfortunately this makes `MysqlOptions` not copyable, so the diff also contains lots of "clones".
Reviewed By: ahornby
Differential Revision: D25055819
fbshipit-source-id: 21f7d4a89e657fc9f91bf22c56c6a7172fb76ee8
Summary:
In the next diff I'm going to add Mysql connection object to `MysqlOptions` in order to pass it down from `MononokeAppData` to the code that works with sql.
This change will make MysqlOptions un-copyable.
This diff fixed all issues produced by the change.
Reviewed By: ahornby
Differential Revision: D25590772
fbshipit-source-id: 440ae5cba3d49ee6ccd2ff39a93829bcd14bb3f1
Summary:
benchmark_filestore XDB subcommands uses mysql and has option of using either myrouter or mysql. In this diff I used `args::parse_mysql_options` function to parse the arguments instead of manual processing and get a `MysqlOptions` object.
This is needed later to pass a connection pool object through the `MysqlOptions` struct (see the next diff).
Reviewed By: ahornby
Differential Revision: D25587898
fbshipit-source-id: 66fcfd98ad8f3f9e285ca9635d8f625aa680d7ff
Summary:
Like it says in the title. This is nice to do because we had old futures
wrapping new futures here, so this lets us get rid of a lot of cruft.
Reviewed By: ahornby
Differential Revision: D25502648
fbshipit-source-id: a34973b32880d859b25dcb6dc455c42eec4c2f94
Summary:
This was kinda almost done. Might as well finish it by updating what's left,
i.e. the tests.
Reviewed By: ahornby
Differential Revision: D25498799
fbshipit-source-id: 65b7b144f5cf86d5f1754f5c7dafe373173b5ece
Summary: Let's not spawn too many futures at once
Reviewed By: markbt
Differential Revision: D25612069
fbshipit-source-id: e48901b981b437f66573a1abfba08eb144af2377
Summary: Forgot to add them when I wrote the test. Let me add tem now
Differential Revision: D25611802
fbshipit-source-id: 0db7bee2034ad6e1566c5eb6de2e80e18140d757
Summary: Convert all BlobRepoHg methods to new type futures
Reviewed By: StanislavGlebik
Differential Revision: D25471540
fbshipit-source-id: c8e99509d39d0e081d082097cbd9dbfca431637e
Summary:
configs.allowedlocations restricts what configs can be loaded to a
certain set of files. This will enable us to deprecate all old config locations.
This diff adds Python support and a high level test.
Reviewed By: quark-zju
Differential Revision: D25539736
fbshipit-source-id: fa2544379b65672227e0d9cf08dad7016d6bbac8
Summary:
We want to start disallowing non-approved config files from being
loaded. To do that, let's update the config verifier to accept an optional list
of allowed locations. If it's provided, we delete any values that came from a
disallowed location.
This will enable us to prune our config sources down to rust configs,
configerator configs, .hg/hgrc, and ~/.hgrc.
Reviewed By: quark-zju
Differential Revision: D25539738
fbshipit-source-id: 0ece1c7038e4a563c92140832edfa726e879e498
Summary:
The goal of this stack is to start logging commits to scribe even if a commit was
introduced by scs create_commit/move_bookmark api. Currently we don't do that.
Initially I had bigger plans and I wanted to log to scribe only from bookmarks_movement and remove scribe logging from unbundle/processing.rs, but it turned out to be trickier to implement. In general, the approach we use right now where in order to log to scribe we need to put `log_commit_to_scribe` call in all the places that can possibly create commits/move bookmarks seems wrong, but changing it is a bit hard. So for now I decided to solve the main problem we have, which is the fact that we don't get scribe logs from repos where bookmarks is moved via scs methods.
To fix that I added an additional option to CreateBookmark/UpdateBookmark structs. If this option is set to true then before moving/creating the bookmark it finds all draft commits that are going to be made public by the bookmark move i.e. all draft ancestors of new bookmark destinationl. This is unfortunate that we have to do this traversal on the critical path of the move_bookmark call, but in practice I hope it won't be too bad since we do similar traversal to record bonsai<->git mappings. In case my hopes are wrong we have scuba logging which should make it clear that this is an expensive operation and also we have a tunable to disable this behavioiur.
Also note that we don't use PushParams::commit_scribe_category. This is intentional - PushParams::commit_scribe_category doesn't seem useful at all, and I plan to deprecate it later in the stack. Even later it would make sense to deprecate PushrebaseParams::commit_scribe_category as well, and put commit_scribe_category optoin in another place in the config.
Reviewed By: markbt
Differential Revision: D25558248
fbshipit-source-id: f7dedea8d6f72ad40c006693d4f1a265977f843f
Summary:
Those messages like "pulling from ...", "added n commits ..." belong to stderr.
This makes it possible for us to turn on verbose output for auto pull, without
breaking tools that parses stdout.
Reviewed By: sfilipco
Differential Revision: D25315955
fbshipit-source-id: 933f631610840eb5f603ad817f7560c78b19e4ad
Summary:
It turns out `Arc::ptr_eq` is becoming unreliable, which will cause fast paths
to be not used, and extreme slowness in some cases (ex. `public & nodes`
iterating everything in `public`).
This diff adds an API for an IdMap to tell us its identity. That identity is
then used to replace the unreliable `Arc::ptr_eq`.
For an in-memory map, we just assign a unique number (per process) for its
identity on initialization. For an on-disk map, we use the type + path to
represent it.
Note: strictly speaking, this could cause false positives about
"maps are compatible", because two maps initially cloned from each other
can be mutated differently and their map_id do not change. That will
be addressed in upcoming diffs introducing a more complex but precise way to
track compatibility.
Reviewed By: sfilipco
Differential Revision: D25598076
fbshipit-source-id: 98c58f367770adaa14edcad20eeeed37420fbbaa
Summary:
On Windows, platform.version appears to be misleading as it returns 1903 for
Windows 10 1909... The registry has the correct build, so let's use that instead.
Reviewed By: kmancini
Differential Revision: D25577939
fbshipit-source-id: f47032906d02669bd1cb1a48304733c1e3499f81
Summary:
With selectivepull we can tell visibility directly about which heads to add,
instead of adding all nodes in a changegroup.
Note: This does not affect the pull command, nor exclude public commits yet.
Reviewed By: markbt
Differential Revision: D25562090
fbshipit-source-id: aa5f346f33058dfdb3b2f23f175e35b5d3c30a1d
Summary: They will be used in core later.
Reviewed By: markbt
Differential Revision: D25562093
fbshipit-source-id: 4402a629a09920fd4c6f85cb8e777446bb218a37
Summary:
When tailing to fill or backfill derived data, omit checking the heads from the
previous round of derivation, as we know for sure they've been derived.
Reviewed By: krallin
Differential Revision: D25465445
fbshipit-source-id: 384c7e67e99c561ce6aae324070e7c274c56b736
Summary:
Rustfmt gives up on formatting if strings are too long. Split the long help
strings so that the formatter works again.
Tidy up some of the help text while we're at it.
Reviewed By: krallin
Differential Revision: D25465443
fbshipit-source-id: 360dbedc1e3e2ffbc489a9d9cba008835bce506f
Summary:
ChangesetInfo derivation doesn't depend on the parent ChangesetInfo being
available, so we can derive them in batches very easily.
Reviewed By: krallin
Differential Revision: D25470721
fbshipit-source-id: cc8ce305990eb6c9846158f0e9e3917cf35e169d
Summary:
Add documentation comments to the derived data crate to describe how they fit
together.
Reviewed By: krallin
Differential Revision: D25432449
fbshipit-source-id: b62440bcecae900ad75d74245ce175bd9e07a894
Summary:
Using `backfill-all` on very large repositories is slow to get started and slow
to resume, as it must traverse the repository history all the way to the start
before it can even begin.
Make this more usable by using the skiplist index to slice the repository into
reasonably sized slices of heads with the same range of generation numbers.
Each slice is then derived in turn. If interrupted, derivation can continue
at the next slice more quickly.
Reviewed By: krallin
Differential Revision: D25371968
fbshipit-source-id: f150ea847f9fbbe84852587d620ae37ba2c58f28
Summary:
Right now, when we upload a hg commit, we check that we have all the content
the client is referencing.
The only problem is we do this by checking that the filenodes the client
mentions exist, but the way we store filenodes right now is we write them
concurrently with content blobs, so it is in fact possible to have a filenode
that references a piece of content that doesn't actually exist.
That isn't quite what one might call satisfactory when it comes to checking the
content does in fact exist, so this diff updates our content checking
In practice, with the way Mononoke works right now this should be quite free:
the client uploads everything all the time, and then we check later, so this
will just hit in the blobstore cache.
In a future world where clients don't upload stuff they already know we have,
that could be slower, but doing it the way is we do it before is simply not
correct, so that's not much better. The ways to make it faster would be:
- Trust that we'll hit in cache when checking for presence (could work out).
- Have the client prove to us that we have this content, and thread it through.
To do the latter, IMO the code here could actually look at the entries that
were actually uploaded, and not check them for presence again, but right now we
have a few layers of indirection that make this a bit tricky (technically, when
`process_one_entry` gets called, that means "I uploaded this", but nothing in
the signature of any of the functions involved really enforces that).
Reviewed By: StanislavGlebik
Differential Revision: D25422596
fbshipit-source-id: 3cf34d38bd6ed1cd83d93c778f04395c942b26c0
Summary:
It always takes a bit of time to find the logs. Since we do have scm daemon onces in the output of `hg cloud status`,
it would be nice to have the onces from background backup as well.
Reviewed By: markbt
Differential Revision: D25560145
fbshipit-source-id: cdf5d76c7c3ebb1492559d32935f9301452a1cd5
Summary: The order has been incorrect and led to a confusing message
Reviewed By: krallin
Differential Revision: D25559963
fbshipit-source-id: 4fcb3e53cedcb08675b60b25cbb5da2ca52c08ed
Summary:
Add a resolve_repos function to cmdlib::args for use from jobs that will run for multiple repos at once.
Planning to use this form the walker to scrub multiple small repos from single process
Differential Revision: D25422755
fbshipit-source-id: 40e5d499cf1068878373706fdaa72effd27e9625
Summary:
In a future diff, paths will be validated to make sure they are valid utf8. The
path sanity checker needs to be constexpr to construct global paths, but the
utf8 functions aren't, so let's write one that is.
Reviewed By: chadaustin
Differential Revision: D25562681
fbshipit-source-id: e48ec835c2cc9dc01090918cc7ee8f61b6c05a20
Summary:
When compiling with Buck, it tries to create hardlink which are rightfully
failing, but that error then triggers a spurious notification. For now, let's
only notify the user on timeout, as these are very likely to be network issues.
Reviewed By: kmancini
Differential Revision: D25500364
fbshipit-source-id: 95b609ae901fa6207c8edba26cd8e6a21ddfe3ac
