Summary:
Use the new memctx.mirror and memctx.__setitem__ APIs. This simplifies the
code.
Reviewed By: DurhamG
Differential Revision: D26726474
fbshipit-source-id: 044616137b883ca250e6d84c0ecfcc70458ec07a
Summary:
Use the Rust tree matcher to rule out files that do not need dirsync quickly.
This should make codemod faster to commit.
I created 5000 files outside dirsync config in fbsource (with 494 lines of
dirsync config), and `hg add` them:
$ mkdir unused
$ cd unused
$ for f in `seq 5000`; do touch $f; done
$ hg add .
Baseline "status":
In [3]: %time repo.status();1
CPU times: user 111 ms, sys: 10.2 ms, total: 122 ms
Wall time: 148 ms
Before, dirsync overhead is ~8x "status":
In [1]: %time x.dirsync.dirsyncctx(repo[None])
CPU times: user 1.37 s, sys: 28.8 ms, total: 1.4 s
Wall time: 1.79 s
Out[1]: (<workingctx f23d7c84c5a7+>, set())
In [2]: %time x.dirsync.dirsyncctx(repo[None])
CPU times: user 1.07 s, sys: 8.41 ms, total: 1.08 s
Wall time: 1.11 s
Out[2]: (<workingctx f23d7c84c5a7+>, set())
After, dirsync overhead is ~1/2 of "status":
In [1]: %time x.dirsync.dirsyncctx(repo[None])
CPU times: user 203 ms, sys: 18.9 ms, total: 222 ms
Wall time: 245 ms
Out[1]: (<workingctx 8ff14e46c9d8+>, set())
In [2]: %time x.dirsync.dirsyncctx(repo[None])
CPU times: user 154 ms, sys: 24.1 ms, total: 178 ms
Wall time: 202 ms
Out[2]: (<workingctx 8ff14e46c9d8+>, set())
Reviewed By: DurhamG
Differential Revision: D26726476
fbshipit-source-id: e34218769c779c9a4ee64c654c75298b7c79f213
Summary: Now dirsync works with IMM rebase. Add a test for it.
Reviewed By: DurhamG
Differential Revision: D26726478
fbshipit-source-id: 6712538d7e903ddb0e3c3df44f7dde638276e99d
Summary: Now dirsync works with absorb. Add a test for it.
Reviewed By: DurhamG
Differential Revision: D26726477
fbshipit-source-id: 4505ad6c1e1fd03bfb2cf12b46bd07c98f2bcc2b
Summary:
Previously, dirsync wraps `repo.commit`, requires an on-disk working copy,
and dirstate to work properly. This diff updates dirsync to wrap
`repo.commitctx` instead, do commit edits purely in memory, then sync the
commit back to disk. It makes dirsync compatible with absorb and in-memory
rebase (and potentially other things like drawdag, if drawdag's context APIs
are improved).
To sync the changes made in-memory back to the filesystem, a dirstate callback
is added to write back mirrored files from commit to disk. This works for both
amend and absorb so the special wrapper about amend is dropped. It is
also optimal for absorb, because it only writes the mirrored files once for
the "final" commit, instead of writing the files for each commit in the stack.
Some `O(N^2)`s (N: len(status)) complexities were avoided:
- `applytomirrors` was called N times.
- `allchanges = set(status.modified + status.removed + status.added)` in
`applytomirrors` was O(N).
- `sourcepath in status.removed` in `applytomirrors` was O(N).
- `mirrorpath in status.removed` in `applytomirrors` was O(N).
Note there is still a suboptimal complexity of `getmirrors` called per changed
path, `O(N*M)` (N: len(status), M: len(dirsync_conig)). That will be addressed
in a later diff.
Reviewed By: DurhamG
Differential Revision: D26726479
fbshipit-source-id: 482c6c830ab65cc0d9cd569a51ec610a1dac49cc
Summary: This is unused, and it has a hard dependency on Fuse, so let's remove it.
Reviewed By: chadaustin
Differential Revision: D26742212
fbshipit-source-id: 091556be39e599512d34920503083d03d4c5a0c2
Summary:
## Why this diff
we want hostname prefix to support targeting configs at clients in corp ("corp" means laptop, labs, and other machines that are not in "prod" datacenters), like FRL machines, that don't support our existing tier mechanism.
## Changes
* Extract hostname prefix in `dynamicconfig.rs` and add a getter function `hostname_prefix()` for it.
*A hostname prefix only consists of alphabetical letters and dashes, which is followed by one or more digits in the hostname. If no valid match, the prefix is set to the empty string.*
* Use `gen.hostname_prefix()` in the `evaluate()` fn inside `mod.rs` to check the generator's prefix against a list of given prefixes.
* Copy changes from `configerator/source/scm/hg/hgclientconf/hgclient.thrift` to `fbsource/fbcode/configerator/structs/scm/hg/hgclientconf/hgclient.thrift`.
* Rebuild in `eden/scm/`.
Reviewed By: DurhamG
Differential Revision: D26706686
fbshipit-source-id: 725506a1c1f0983e981b0b3f3993c7c14510b1db
Summary:
dynamicconfig has logic to avoid loading remote configs during tests,
by checking the TESTTMP variable. Unfortunately, the tests currently run system
hg with that environment set, which causes it to generate incorrect
configuration for the real repository (i.e. fbsource).
Let's get rid of all test specific environment variables when running the system
hg. There's a comment in the code saying this breaks some tests (from 2017
03a31be593db), but as far as I can see they still work.
Reviewed By: quark-zju
Differential Revision: D26758364
fbshipit-source-id: 945cbd36f7eccc98149772a42421cbc51b157c5b
Summary:
The new test case failed on Mac because it didn't want to copy over a
symlink or something. Didn't dig too deep because 'rm -rf' works around it.
Reviewed By: sfilipco
Differential Revision: D26758985
fbshipit-source-id: 22ad7e23813bd6bfe86b91535d46c21ee59dfcd1
Summary:
`nonfilelogfctx` means the `fctx` does not have a backing filelog, is in-memory.
Previously, one of the `fctx` is assumed to be backed by a concrete filelog,
it will crash if that is no satisfied.
Reviewed By: DurhamG
Differential Revision: D26731810
fbshipit-source-id: f758f43dc140dbf6cc9b6234681c4830482e767a
Summary:
In a corner case, `mctx[f]` is `None`, and `f` does not exist in both parents.
The `f` should not appear in te `added` status list.
Fix it and remove a duplicated call to `managing`.
Reviewed By: DurhamG
Differential Revision: D26726482
fbshipit-source-id: f386ad9eb27bf053fb671e0f187aaada4699e84e
Summary:
This is used by next diffs.
Test code:
In [3]: v=repo[None]['.gitignore']
In [4]: v.cmp(v)
Before:
RuntimeError: maximum recursion depth exceeded
After:
Out[4]: False
Reviewed By: DurhamG
Differential Revision: D26726481
fbshipit-source-id: c27a8f414313f262d0f596b058df580f9216e8fc
Summary:
Not all context objects support the `__setitem__` API. Instead of adding supports
for all of them, let's add an API to create a `memctx` that mirrors the changes
from another `ctx` so one can them do mutations on the `memctx`. This will be
used by dirsync.
This might also be useful in other "amend" cases, because the `mirror` API takes
various kinds of context (workingctx, changectx, memctx), and is straihtforward
to use:
In [2]: v=m.context.memctx.mirror(repo['.'])
In [3]: v.commit()
Out[3]: 'b\xdeF\xbb\xbb&\xe5\xdbvJ\x16\xf2\xc3\x92\x96Mr\xb7\x85\xbb'
In [4]: repo['.'].node()
Out[4]: 'b\xdeF\xbb\xbb&\xe5\xdbvJ\x16\xf2\xc3\x92\x96Mr\xb7\x85\xbb'
In [5]: v['x']=v['.gitignore']
In [6]: v.commit()
Out[6]: 'b\xdeF\xbb\xbb&\xe5\xdbvJ\x16\xf2\xc3\x92\x96Mr\xb7\x85\xbb'
Reviewed By: DurhamG
Differential Revision: D26726480
fbshipit-source-id: 629cfc721f79d29ef395d4e1d6b3381f0095d573
Summary:
Add `memctx.__setitem__(path, fctx)` API to change a file in memctx. This will
be used in upcoming dirsync changes.
Regarding on `memctx` vs `overlayworkingctx`. Both need some changes to fit the
dirsync use-case (overlayworkingctx needs "amend", not "commit on top of",
memctx needs to be mutable).
The main difference is `overlayworkingctx` uses a concrete `cache` dict for
path, data, and metadata. `memctx` uses a concrete list of file paths, but file
data, metadata is abstracted - it uses a function that returns fctx. I would
like to preserve the laziness of reading files and metadata (especially for LFS),
so I picked the `memctx` instead of `overlayworkingctx` as the starting point.
Reviewed By: DurhamG
Differential Revision: D26726475
fbshipit-source-id: 7c16980ffa1c83fe0ff6d70c60b1daae5e266683
Summary: This simplifies the code a bit.
Reviewed By: kulshrax
Differential Revision: D26681779
fbshipit-source-id: 393565790ab711dd09ae6cfa6f9c4b19c930eb93
Summary:
Similar to D26670318, use the EventListeners APIs to implement the progress
callback. Now HandlerExt only has RequestContext related "as_ref" logic.
Reviewed By: kulshrax
Differential Revision: D26681778
fbshipit-source-id: b7f6e07ced43e0ae043e859337c06b69bd5dfc95
Summary: This makes it useful in non-mut callbacks.
Reviewed By: kulshrax
Differential Revision: D26681784
fbshipit-source-id: 97312df8bf3f900a36cbeb27206a2946bb6c702c
Summary: This makes ProgressUpdater Send + Sync so it can be used in the new callback APIs once `mut` gets dropped.
Reviewed By: kulshrax
Differential Revision: D26681781
fbshipit-source-id: 9c622b1d78b4091e3359c28972b6624f0b53734d
Summary:
This removes more mutable fields. Note the new code is more correct because
curl can call the `progress` callback periodically even if no progress is made.
According to https://curl.se/libcurl/c/CURLOPT_PROGRESSFUNCTION.html:
This function gets called by libcurl instead of its internal equivalent with
a frequent interval. While data is being transferred it will be called very
frequently, and during slow periods like when nothing is being transferred it
can slow down to about one call per second.
Reviewed By: kulshrax
Differential Revision: D26681780
fbshipit-source-id: 19aa4bcb4c56623e3f0408b06041b3a894f197e7
Summary:
This makes the `total_progress` field use lock-free interior mutability. The
goal is to eventually drop Rc and RefCell.
Reviewed By: kulshrax
Differential Revision: D26681782
fbshipit-source-id: ec0a6abbb2115c17c674db2255d196aaec847705
Summary: This removes the need for `mut` for this field.
Reviewed By: kulshrax
Differential Revision: D26681783
fbshipit-source-id: 10ed9adfb62081b0e6839abd9534db92d4e056c5
Summary:
The ProgressInner only exposes APIs for total (aggregated) progress. There is
no API to read individual progress. Make it only track the total progress. This
would make it easier to implement interior, lock-free mutability on ProgressInner.
The updater now needs `mut` temporarily, which will be dropped later (D26681784).
A test case is tweaked so progress does not go backwards.
Reviewed By: kulshrax
Differential Revision: D26681777
fbshipit-source-id: 4ad1b9173d5a2c2326e00c030d51f77e9b9458f3
Summary: Test both the HttpClient and Request events.
Reviewed By: kulshrax
Differential Revision: D26670325
fbshipit-source-id: ffbc4268f7de698830411434a769c8b1a4acd863
Summary:
This simplifies the code a bit and makes it look consistent with other event
listeners.
Reviewed By: kulshrax
Differential Revision: D26670318
fbshipit-source-id: f6eda9403bb6eb09a074544e672a45c84f38e2b1
Summary:
Add `RequestContext.event_listeners()` to register callbacks, and trigger the callbacks
when related events happen.
Reviewed By: kulshrax
Differential Revision: D26670323
fbshipit-source-id: 9b92b715444e83957c06b06f1ce696d4de3c0023
Summary:
This simplifies the logic a bit. There is no need for
`HandlerExt::with_payload` that is similar to `Request::body`.
Reviewed By: kulshrax
Differential Revision: D26670326
fbshipit-source-id: 9fe755821062ad6f2a74d6d5ba345620669f0f63
Summary:
This diff adds a gflag option to turn on `TreeOverlay`. As this diff, this
option only works on Windows.
Reviewed By: xavierd
Differential Revision: D25223471
fbshipit-source-id: ccab4c85cf4b08980e67814238457a8eb83af3dc
Summary:
This diff adds `TreeOverlay` that implements the `IOverlay` interface based on
`TreeOverlayStore`. At this point this class does not do much other than
forwarding calls to `TreeOverlayStore`. This layer will become necessary in the
future when we add file handling support to `TreeOverlay`.
Reviewed By: xavierd
Differential Revision: D25223473
fbshipit-source-id: 02d03081ba10c82123fe69c35dbc70f865c8370d
Summary:
This diff adds `TreeOverlayStore` that is used in following diffs to store tree
data in SQLite database. This class encapsulates the SQL statments needed for
storing trees.
Reviewed By: chadaustin
Differential Revision: D24083787
fbshipit-source-id: 61c08617fe50e7eac94c614948f5ba5355dba71a
Summary:
We don't need to care if a directory from overlay exists or just empty. This saves the overlay implementation from keeping track of what directories are stored and which are not.
This diff changes `Overlay::loadOverlayDir` to return `DirContents` instead of `std::optional`. The same method on `IOverlay` remain unchanged to give the backing overlay implementation the freedom to choose.
Reviewed By: chadaustin
Differential Revision: D25507406
fbshipit-source-id: f7edcc55485fabeedfe11e9f269eea15a3cc32ad
Summary:
This diff adds more logging to SQLite so we can see what statements are being
executed in logs. Setting `eden.fs.sqlite=DBG9` will show every SQL statement
being executed and bound values.
Reviewed By: chadaustin
Differential Revision: D25223472
fbshipit-source-id: ed089b7ec112c75d9f7bc63d9fe53f0ec2bd6420
Summary:
This diff introduces `transaction()` to `SqliteDatabase` to make managing SQLite transaction easier.
Usage:
```
db->transaction([](auto& txn) {
SqliteStatement(txn, "SELECT ...");
SqliteStatement(txn, "INSERT ...");
});
```
It will automatically roll back when exception happens, and commit when the closure finishes.
Reviewed By: chadaustin
Differential Revision: D25102179
fbshipit-source-id: 5a4ced8c6eb8016e15b8132be2b5abcc9760e2f9
Summary: This diff adds an alias `SqliteDatabase::Connection` to make it less mouthful when we want to reference the type of the SQLite connection we created
Reviewed By: xavierd
Differential Revision: D25386777
fbshipit-source-id: a6a58d522200fe0bf35ac52f6dad16063c0389d0
Summary:
To support the new _Relational Overlay_, we need to know what is actually being
modified by overlay. This diff adds two method `addChild` and `removeChild` to
IOverlay that supports passing such information to the actual implementation.
Reviewed By: chadaustin
Differential Revision: D24153544
fbshipit-source-id: 04e5f4c231ff9b5fe0ee5be082b1a75d78758bb4
Summary:
This is needed for the next diff in the stack to build on Windows. We never
build anything form `./inodes/overlay` on Windows.
Reviewed By: xavierd
Differential Revision: D24258241
fbshipit-source-id: e1c5234433a1ec92419dcb090c8949800efa605f
Summary:
This diff introduce `IOverlay` that provides a shared interface between
`FSOverlay` and `SqliteOverlay`. This allows us to share more code between
POSIX and Windows, and prepare for the new SQLite-backed Overlay
implementation.
Note as of this diff `IOverlay` is only at a very initial version. The
interface is far from complete and still need a few iteration to change.
Reviewed By: chadaustin
Differential Revision: D23941454
fbshipit-source-id: 69cce9d3852f21b2d19f528e7e05fb183538ace9
Summary:
This diff rollouts V2 of autocargo in an atomic way so there are quite a few things done here.
Arc lint support:
V1 used to be part of the default fbsource `arc lint` engine, but since V2 calls buck it must live in a separate lint engine. So this diff:
- Adds running `autocargo` as part of `arc lint-rust`
Mergedriver update:
- Mergedriver used in resolving conflicts on commits is now pointing to V2
- It handles files in `public_autocargo/` directories in addition to the ones containig generation preamble
Including regeneration results of running `common/rust/cargo_from_buck/bin/autocargo`. All the differences are accounted for:
- Some sections and attributes are removed as they can be autodiscovered by Cargo (like `lib.path = "src/lib.rs"` or empty [lib] section)
- "readme" attribute is properly defined as relative to Cargo.toml location rather than as hardcoded string
- "unittest = false" on a Buck rule propagates as "test = false; doctest = false" to Cargo
- "rusqlite" is not special-cased anymore, so the "budled" feature will have to be enabled using custom configuration if required by the project (for rust-shed in order to not break windows builds a default feature section was added)
- Files generated from thrift_library rules that do not support "rust" language are removed
- Custom .bzl rules that create rust artifacts (like `rust_python_extension`) are no longer ignored
Others:
- Changed `bin/cargo-autocargo` to be a wrapper for calling V2 via `cargo autocargo`
- Updated following files to use V2:
- `common/rust/tools/reindeer/version-bump`
- `remote_execution/rust/setup.sh`
- Removed few files from V1 that would otherwise interfere with V2 automatic regeneration/linting/testing
Reviewed By: zertosh
Differential Revision: D26728789
fbshipit-source-id: d1454e7ce658a2d3194704f8d77b12d688ec3e64
Summary:
Changes that could be done as a separate step from the rollout diff:
- Added README.md with documentation for developers
- Pointed `external_linter_tests/run_linter.sh` to MSDK/dotslash released autocargo
- Added `project_configs/ios_static_listing.toml` to reflect changes in `autocargo_projects.py` that were made in the meantime
- Added `scripts/run_tests` to run `arc external-linter-test` using cargo built autocargo binary
- Modified `third-party/rust/Cargo.toml` and run cargo-autocargo to match formatting of thirt party crates produced by V2
Reviewed By: ahornby
Differential Revision: D26728788
fbshipit-source-id: 361d9e2d41685f34eb4d20fc5ad9930f24b48f85
Summary:
Previously we were just logging whether derivation was successful or not.
Let's start logging the error as well.
Reviewed By: krallin
Differential Revision: D26750165
fbshipit-source-id: 03983cfa7bfbd5aeeed27ffac5a7df96c05e0967
Summary: Ordering is flaky, so sorting it.
Reviewed By: StanislavGlebik
Differential Revision: D26726727
fbshipit-source-id: dd60da022fe6fd2568be5b3108edc399c8db390c
Summary:
With the upcoming rollout of C++ Manifold client, it's useful to know which blobstores are which in our Scuba log, `mononoke_blobstore_trace`.
Pass a debug name down to help
Reviewed By: krallin
Differential Revision: D26673162
fbshipit-source-id: e7c16ad217d8daf21565939a45ac82204459055a
Summary:
Add a way to define event listeners. Define events on HttpClient and
Request(Info). They will be used by upcoming changes.
The idea is similar to GUI programming. A "control" has a list of "events" that
handlers can be defined on them.
This diff defines lists of events on the "client" and "request" types.
Areas this diff tries to improve:
- Make it easier to add new events (by using macro_rules).
- Make it easier to visually see all possible events.
Reviewed By: kulshrax
Differential Revision: D26670324
fbshipit-source-id: 92f74779f8e546491d0e922db27a4b87f527a5e9
Summary:
This makes it easier to access states defined in `RequestContext` from
`curl::Easy2<H>` types.
Reviewed By: kulshrax
Differential Revision: D26670317
fbshipit-source-id: 27eca9dcc11b14b5d41c8327448f7748ebc62e10
Summary:
Upcoming diffs extend the trait with new methods unrelated to configuration.
Rename to clarify.
Reviewed By: kulshrax
Differential Revision: D26670314
fbshipit-source-id: 7d33ebe22b26f1a286ae40c78f51f31a1a64957e
Summary:
Similar to the Streaming curl handler, add a RequestContext field to the Buffered
handler so the curl callbacks on the Handler can provide the Request
information like urls.
Reviewed By: kulshrax
Differential Revision: D26670321
fbshipit-source-id: de7abecf162c4aaed03d927c35516b6f8ac523ce
