Summary:
Previously the hg serve command requires a chain of other commands to run:
hg (client) -> /bin/sh -> /usr/bin/env -> python -> dummyssh -> hg (server)
This diff makes it run the server process directly:
hg (client) -> hg (server)
So there is no dependency on /bin/sh (or cmd.exe), a standalone python etc.
The dummyssh logic is "inlined" so the stdout + stderr order is preserved.
Test changes:
- run-tests and testutil: ensure sys.argv[0] is set to hg, not the t.py test
so util.hgcmd() works
- test-clone, test-pull, test-push: removed checks about ssh injection - we
no longer use ssh in production anyway.
- test-fb-hgext-pushrebase-remotenames: minor output update
- test-commitcloud-sync: incompatible and irrelevant --bgssh test removed
- test-infinitepush-bundlestore: irrelevant and removed
- test-rustmanifest-t: stdout / stderr order adjusted
Reviewed By: DurhamG
Differential Revision: D34835693
fbshipit-source-id: ff71ec1cfe57ea9b00a811703b89211b1b8fc76a
Summary:
TestTmp sets up the test environment like what run-tests.py does per test.
It does not affect global state so one can use TestTmp in multiple tests
in one single test file.
See the docstring of the added TestTmp class for examples.
Reviewed By: DurhamG
Differential Revision: D34725132
fbshipit-source-id: 422b73adb82b0373868a776cf703ca36d6cd6c52
Summary: This is an attempt to bypass some slow code in our logic to revert so that it works fast more consistently.
Reviewed By: mrkmndz
Differential Revision: D35782843
fbshipit-source-id: 8ec956d37e8f62537ff9f8d29e0a0fadff405f00
Summary:
This makes it easier to use Python scripts to check and view contents of
indexedlog.
Reviewed By: DurhamG
Differential Revision: D35585546
fbshipit-source-id: cd2022e4d6d67a18d43052fbccfd1580721ef23c
Summary:
Use [cog](https://nedbatchelder.com/code/cog) to generate code based on files.
This makes it easier to update the module list.
Reviewed By: DurhamG
Differential Revision: D35585547
fbshipit-source-id: 60f3c9a5bde2a1460ebe64a5701693f60f9ccd13
Summary: Make it easier to add a new module to bindings.
Reviewed By: DurhamG
Differential Revision: D35585550
fbshipit-source-id: 9aade63c586a69413f977dccced030ebddddb039
Summary: It is no longer used after D29020190 (8cc60851cb).
Reviewed By: DurhamG
Differential Revision: D35585549
fbshipit-source-id: 8640e790b1c2f3b2a7ade7ccfe2711bea98bba4b
Summary:
I forgot to add `#[clap(long)]` to the `--storage-id` arg and also the tw config was set to pass `--storage-id` as a main command arg while it's actually a sumbcommand arg. Because of that the walker tasks were failing with :
```
rror: Found argument '--storage-id' which wasn't expected, or isn't valid in this context
If you tried to supply `--storage-id` as a value rather than a flag, use `-- --storage-id`
USAGE:
walker --scuba-dataset <SCUBA_DATASET> --mysql-query-time-limit <MYSQL_QUERY_TIME_LIMIT> --fb303-thrift-port <PORT> --blobstore-read-qps <BLOBSTORE_READ_QPS> --manifold-request-priority-override <MANIFOLD_REQUEST_PRIORITY_OVERRIDE> --blobstore-scrub-grace <BLOBSTORE_SCRUB_GRACE> --blobstore-scrub-action <BLOBSTORE_SCRUB_ACTION> <--repo-id <REPO_ID>|--repo-name <REPO_NAME>> <--config-path <CONFIG_PATH>|--config-tier <CONFIG_TIER>|--prod> <SUBCOMMAND>
```
Reviewed By: markbt
Differential Revision: D35783175
fbshipit-source-id: 5efe68e78730212dfa02db3f17f071b9a4bff490
Summary: Current console output includes only high level snapshot update/restore data. This diff adds instrumentation over the individual steps within the snapshot update/restore command.
Reviewed By: yancouto
Differential Revision: D35750512
fbshipit-source-id: 0d7dfb9550f5fb72ecaf08091e5a5501d652bfe4
Summary:
This diff replaces old cmdlib walker target with the new one. It also changes the TW config because scuba logging arg name now different.
I will stop the conveyor from the updates before I land these changes.
Reviewed By: markbt
Differential Revision: D35532342
fbshipit-source-id: e243d9ce11ea25061812b4021a575e8f57e828c4
Summary:
TL;DR: File invalidation after checkout is broken on NFS macOS. This proposes a
fix.
Previously, to invalidate things on NFS we opened and closed all the parent
directories of any files/directories that changed during a checkout operation.
This worked on Linux because all open calls result in some request into the
EdenFS daemon (usually a getattr I think). The returned response from this
would show that the directory had update timestamps. So the open would see the
parent directories in their updated state. This would trigger the NFS client to
clear it's caches for that directory and their recursive children to preserve
CTO. CTO or close-to-open consistency guarantees that if on process observed a
file in state A and closed the file, then another process opened that same file
, it will see the file in state A or a later state.
macOS does not seem to do CTO.
It seems that most of the "read" filesystem calls can basically be served from
cache on NFS. Only writes seem to be guaranteed to make it into eden.
So instead of using a "read" filesystem call to trigger invalidation, we need to
use a write one.
I tried opening things in write mode, but the problem is that we need to
invalidate directories (to ensure the entry information is updated) and
directories can not be opened in write mode.
I tried writing 0 bytes to the files themselves that have changed, but this
empty write is short circuited somewhere in the kernel.
The only filesystem call that can be a noop, and seems to trigger a call into
eden is chmod. We are not really working off any guarantees any more, but
it seems to work on both Linux and macOS in my testing so far and is better
than our current broken state of invalidation.
So now to invalidate we chmod parent directories that have changed with their
current mode. Note that this could get extra tricky if we were mixing updating
directory permissions with invalidating them. We would need to ensure we chmod
with the most up to date mode bits. However, because we do not update
permissions anywhere that we also invalidate (checkout does not know how to
update directory permissions) things are a little simpler.
It's important that the chmod can not complete until it seems an updated view of
the parent directory. So we have to be careful about locking here. (Although
that hasn't really changed since we switched from open to chmod.)
One thing that does change is that since chmod is technically a write, it could
cause extra materialization that would be bad for checkout and status
performance. To prevent these problems I have updated setattr to not materialize
a directory when the attribute change is a noop which it should be in our
invalidation technique unless another client modified the directory in the
meantime in which case the directory should be modified anyways. This would
mean that we could end up wiping out clients changes to permissions in the
working copy during checkout. but this matches non eden checkout behavior. So I
think this is ok.
Reviewed By: xavierd
Differential Revision: D35435764
fbshipit-source-id: 196c4995b130b595f9582204237e726e5212993f
Summary: Forgot to update these when I changed changed the signature.
Reviewed By: DurhamG
Differential Revision: D35748246
fbshipit-source-id: f1102b60a8dd3d4af8312d7c67a0895691774f6a
Summary:
In Mercurial terminology, a `hg reset` operation simply updates the working
copy parent but doesn't affect the state of the repository. A diff/checkout
operation however would compare the on-disk state of the repository with the
working copy parent. Thus EdenFS must respect this behavior.
On Linux and macOS, a reset operation merely updates some state in the
EdenMount, but since these platforms reads data from the inode hierarchy, the
reset operation doesn't affect future reads to the repository. These would
still read the data from the previously checked out revision.
On Windows however, ProjectedFS requires us to read data from the currently
checked out commit, and thus we bypass the inode hierarchy. The downside is
that a reset operation changes the RootId in the EdenMount which is used to
walk Mercurial's tree hierarchy. What this means is that a read operation
performed after a reset would read the content of the file as it is in the
reset revision. This is not the behavior that Mercurial expects.
To solve this, we need to keep track of both the last checked out revision to
be used in response to read callbacks and the last reset revision, to be used
in diff operations. To achieve this, we can use the newly introduced SNAPSHOT
file format.
Reviewed By: fanzeyi
Differential Revision: D35293079
fbshipit-source-id: 5cfae510db8bc759d51447e6c24f021c104b7353
Summary:
On macOS unlink(dir) results in a PermissionError instead of a
IsADirectoryError.
Reviewed By: DurhamG
Differential Revision: D35756194
fbshipit-source-id: bbdc828f112e600c1dcecedb5165dd3b7a8694b2
Summary:
The test was flaky because the output sometimes is in different order. The test already had `sort` command for the walker output, however the strings that were sorted included much more than just the failed types:
```
# The output has globbing, which consumes the beginning (after the "Validation failed: ") and the end of the string
Validation failed: *hg_link_node_populated* (glob)
# while the actual string looks like this:
Validation failed: {"int":{"check_fail":1,"seq":1,"time":1650377693},"normal":{"build_revision":"","build_rule":"fbcode:eden/mo
nonoke/walker:new_walker","check_type":"bonsai_phase_is_public","node_key":"changeset.blake2.2b06a8547bfe6a3ac79392aef3fa7f3f45a82
f4e0beb95c4fa2b914c34b5b215","node_type":"PhaseMapping","repo":"repo","server_hostname":"testingnamespace","session":"kX8ip3P1uUxV
58Mt","src_node_key":"changeset.blake2.2b06a8547bfe6a3ac79392aef3fa7f3f45a82f4e0beb95c4fa2b914c34b5b215","src_node_type":"Changese
t","via_node_key":"hgchangeset.sha1.26805aba1e600a82e93661149f2313866a221a7b","via_node_type":"HgChangeset","walk_type":"validate"
}}
```
So when the numbers changed in between the test runs the order of the results changed too.
Now let's sort the output properly: only the failed types.
Reviewed By: yancouto
Differential Revision: D35747592
fbshipit-source-id: eddca83bc9f1ecc202f87f0079ce2e65abd0c479
Summary:
The current UX of `eden status` isn't good. It leaks too much internal details and generates confusing error message for the user.
This diff fixes the error reporting by moving the exact detail into trace logging while the actual error is more user-friendly.
This should reduce the confusions.
Reviewed By: chadaustin
Differential Revision: D35663792
fbshipit-source-id: 0b76f9ff9ee4542cce7349ccfa6724f096e1305f
Summary: Eden's `getEntryInformation` API currently loads Inodes for all paths queried, force-materializing the path component's Inodes. This often isn't required, and eden could be fetching non-loaded data from the object store instead.
Reviewed By: chadaustin
Differential Revision: D35372760
fbshipit-source-id: e31a450a20b09249f03339dcd1aeca2eb363046e
Summary: There's no need to materialized iNodes to request the content Sha1 and/or BlobMetadata, as that can be fetched from the objectStore.
Reviewed By: chadaustin, xavierd
Differential Revision: D35467564
fbshipit-source-id: 2848f4d21725a9f5d40251fde2e0eb29ea81302e
Summary: Instead of stripping anyhow context off all errors, only strip within the concrete error handler "specific_error_handler". This way, the fallback error handler (that produces the beloved RustError) gets the top level error with all the anyhow context.
Reviewed By: DurhamG
Differential Revision: D35602091
fbshipit-source-id: 547ae7e1a2352af74817f90ab08cadb0fce65efc
Summary:
Following diff D35738033 adds another crate and reindeer generates unrelated changes.
Run
```
fbcode/common/rust/tools/reindeer/vendor
```
without any changes.
Reviewed By: dtolnay
Differential Revision: D35738034
fbshipit-source-id: 0539b8abbd694479dbe89939cb8c5ddc6272bd71
Summary: This code has enough risk of a copy paste error that it deserves a unit test.
Reviewed By: chadaustin
Differential Revision: D35161787
fbshipit-source-id: 5691d13a74a0f059dfd6a93ea2852dca8399a165
Summary:
Most hghave checks are "immutable" results. Memorize them to reduce cost.
If some checks depend on actual tests, we can special case them later to
not be memorized.
Reviewed By: DurhamG
Differential Revision: D34725137
fbshipit-source-id: f010bddb856df1950854b839aa8c311c6a40bdbd
Summary:
This allows using the hghave library without shelling out to a `hghave` binary
in tests.
Reviewed By: DurhamG
Differential Revision: D34725136
fbshipit-source-id: 1b8115a791414a68ebd2e7ec077f77bca2a0833a
Summary: `sed 's/a/b'` (and other sed commands) are used frequently in tests. Implement it.
Reviewed By: DurhamG
Differential Revision: D34790212
fbshipit-source-id: 76bfac795e6b14a8d83cbee9fd5c74341112786e
Summary:
`grep` is commonly used. Implement it.
This diff might also serve as an example about how to extend the stdlib.
Reviewed By: LynBusch
Differential Revision: D34725127
fbshipit-source-id: 32ba5cdfc521f538923f05c09c2fa9afb6e88f05
Summary:
To be able to execute .t test without bash, and to be future
proof so tests can use Python more easily instead of bash.
Define a way that supports both classic .t and pure Python
and a mix of them.
See the docstring for details.
Not all `.t` features are handled in this diff.
Namely `#testcases` support is missing.
Reviewed By: LynBusch
Differential Revision: D34725133
fbshipit-source-id: f68c88a2e5c0fa4f3ce3de0b39ff17e11350b1be
Summary:
See the added module docstring for details.
Unlike testutil/dott (D16172901 (27908b883c)), this implementation behaves
much more closely to a real shell thanks to conch-parser.
For reviewers: This is a large change but still relatively
simple compared to other shell implementations. It might be
easier just looking at the tests.
Differential Revision: D34725134
fbshipit-source-id: 1ec54ae2c0146ba2533c16f3fbc27f4275ca80f4
Summary:
This makes it easier to export the AST in other formats (ex. Python objects).
Conflicted names (ex. Parameter::Star vs SimpleWord::Star) are attributed with serde(rename) so they can be distinguished more easily.
Reviewed By: LynBusch
Differential Revision: D34725130
fbshipit-source-id: d91a80cec5b858669f5d16c95aa63046470730e1
Summary: Add TARGETS so it can be used in buck builds.
Reviewed By: yancouto
Differential Revision: D34725122
fbshipit-source-id: 00b958738b05a30a4b9a1470978645b5fad0ad8d
Summary: This makes the code compile with future versions of rustc.
Reviewed By: LynBusch
Differential Revision: D34725123
fbshipit-source-id: ec0d4b0e5fee338487dbb817ac64333a6f7a1040
Summary:
Import https://github.com/ipetkov/conch-parser commit
e1d3ef269684f46c9696bf3f83e81c37e711570d under MIT license.
The conch-parser seems to be a decent shell parser that
solves the shell syntax parsing headache. For example,
it handle heredocs with substitution (`<< EOS`, `$V`
should be substituted) or no-substitution (`<< 'EOS'`,
`$V` or a slash does not have special meaning) properly.
Heredoc is frequently used in our tests. Most other
shell parsing libraries I checked lack proper heredoc
handling, and/or have an incompatible license. Although
it seems some corner cases can be improved, for example,
`A=1 for i in $A; do ...; done` treats `for` as a
normal command instead of a shell loop keyword. But
that does affect the main test use-cases.
In a later diff the library is modified to support serialization.
Given that the public crates.io is too old and the modification
needed, I imported it into edenscm's own vendor directory
for local customization.
Differential Revision: D34725119
fbshipit-source-id: 3e7d3d3d4f92380a0fd104275e75863332b429db
Summary: A script to run the benchmark for ```setPathObjectId```
Reviewed By: chadaustin
Differential Revision: D35271074
fbshipit-source-id: a0f6d9ab1fe6ace929f615a620e2cd4c36fdcc0e
Summary:
Part of T117105786 -- a set of test failures that bisect to {D35558601 (238a176019)}.
D35558601 (238a176019) already made these same changes to eden/hg-server/lib/indexedlog/src/base16.rs and eden/scm/lib/indexedlog/src/base16.rs. I didn't imagine that there would be not 2 but **4** identical copies of this data structure and tests. :O
Reviewed By: zertosh
Differential Revision: D35660712
fbshipit-source-id: b2e624da04eafd1b2f3b13a0219d84645d96fcdd
Summary: Merging a repo that has multiple refs / heads / unmerged branches (relatively to master) can be dangerous. Let's make users aware of that.
Reviewed By: farnz
Differential Revision: D35608687
fbshipit-source-id: a103862ebd89c034cfe23b27606aa558cddb069c
Summary: This part of segmented changelog was never tested and we depend on this behaviour for emergencies. We're going to depend on it even more now that we won't reload the SC every hour.
Reviewed By: yancouto
Differential Revision: D35614793
fbshipit-source-id: cd10deceab7fb1be5a77f26ef904c09ea7562996
Summary:
We should not reload and reset the server-side segmented changelog if the
tailer has not written a new version since we last loaded - instead, we should
keep our existing one in-memory, because it will be more up to date than the
one we load.
In situations when the tailer is not working properly this might mitigate the issue for the customers instead of exposing them to it.
Reviewed By: farnz
Differential Revision: D35408139
fbshipit-source-id: 10d30b6d2d055fc938f9d0f22f3c633f7d7495c2
Summary:
We use completely different codepath for testing, let's go through the same as
we go in prod.
Reviewed By: farnz
Differential Revision: D35408141
fbshipit-source-id: 05f3f026c0c15b7b460c554828089247f8680eaf
Summary:
This makes 'buck build mode/win //eden/scm:hg' work on Windows. The
resulting binary doesn't actually run though, since buck run depends on hg.sh
which doesn't work on Windows. That will need to be converted later.
Reviewed By: quark-zju
Differential Revision: D35133291
fbshipit-source-id: 15070930612b56a381a8c8e311f6eaac3f1d9856
Summary:
This diff makes the manager return the loaded segmented changelog version
which is going to allow us
Reviewed By: farnz
Differential Revision: D35408143
fbshipit-source-id: 748901c992e2e67e561970f836892a13c6d88e27
Summary:
To avoid unnecessary reloads we need a facility to check the latest version (so
we can compare it with one currently loaded)
Reviewed By: farnz
Differential Revision: D35408138
fbshipit-source-id: 14021edda52105fad71ce430feee42bc42659546
Summary: This will allow me to augment it with other stuff.
Reviewed By: yancouto, farnz
Differential Revision: D35408140
fbshipit-source-id: 1096255eabd84c995ba0bb71209eafd3974c0610
Reviewed By: zertosh
Differential Revision:
D35615001
Ninja: need to fix autocargo and oss-eden-darwin-getdeps is taking forever
fbshipit-source-id: 1dfc74532615823be3e4cb39ff82e1f7cc0ea1a6
Summary:
By lifting construction of the BackingStores into main, the core
"service" target no longer needs to depend on their
implementations. This shaves almost 60% off the unit test runtime on
my devserver. The tests could be made even faster by decoupling all of
our core logic from HgBackingStore and friends.
Reviewed By: xavierd
Differential Revision: D34774356
fbshipit-source-id: 87c2d5f44cfd84d6e01bea44dd4105f6415ce162
Summary:
Some backing stores have a 1:1 relationship between blobs and their
IDs. Others don't. If they do, checkout can compare blob contents more
quickly.
In addition, add a config knob that we can use to opt hg into the fast
path, at the risk of correctness. It might be useful for getting out
of sticky situations where any update triggers a ton of fetches.
Reviewed By: xavierd
Differential Revision: D34761024
fbshipit-source-id: 36d513d37c38fc29a708e583ebc2614f07f11bb4
Summary:
We currently have no default. Untracked files can be very large (like binaries and such), and easy to miss. We already had an option for that, but were not using it by default. Should the default be even smaller?
It also didn't work with invalid symlinks, as `Path.stat` tried to follow symlinks, so I fixed that.
If a client really wants to upload a large file, they can specify the command line argument by themselves.
Reviewed By: farnz
Differential Revision: D35498620
fbshipit-source-id: 7c21c779a680fb209b82943ec59aa5523cd1608c
Summary:
This diff imports the most recent version of `schema.docs.graphql`
available on https://docs.github.com/. Details explained in the `README.md`
included in this commit.
Reviewed By: quark-zju
Differential Revision: D35600010
fbshipit-source-id: 52050db604ca884ce7465da58c3ed2aa3fb85dbc
Summary:
Currently unit tests that need a segmented changelog create it separately from the repo because in test repo there is only a `DisabledSegmentedChangelog`. I am making it possible to have an `OnDemandUpdateSegmentedChangelog` that is both useful and can be updated to include needed commits.
Look on new `test_is_ancestor` test to see what is made possible.
Reviewed By: markbt
Differential Revision: D35053513
fbshipit-source-id: 221600b50c8a0140b08f633dc3758489f2524f60
Summary: Segmented Changelog requires `CoreContext` for construction, so we can create a test mock if `TestRepoFactory` has a `FacebookInit`.
Reviewed By: markbt, mitrandir77
Differential Revision: D35251021
fbshipit-source-id: 57051e227804669794f19fc2ca130f52cb938b0c
Summary:
This removes the historical check in our Mercurial codebase that
disallows underscores in names (modulo a substantial allowlist).
Incidentally, upstream Mercurial also relaxed this restriction in Oct 2019:
https://phab.mercurial-scm.org/D2010 (40bbe7b4da).
Note it is neither the goal nor the expectation that we start going around
renaming existing members to include underscores. We expect files to
maintain internal consistency, at least as far as their API names are
concerned. Inevitably, code authored under the old convention may
call into other APIs that favor underscores (and vice versa) such that
we cannot demand 100% consistency within a file.
Reviewed By: quark-zju
Differential Revision: D35597630
fbshipit-source-id: 5e3c02ec2e5810aa58123b5e75b59241a4a1bdbb
Summary:
Version 2 profiles are only respected when they are included by the
top-level sparse config. Instead of loading top-level config, debugsparsematch
and debugsparseexplain were loading the profile directly. This meant they didn't
see the version 2 metadata.
Let's fix it by loading a fake top-level config that simply includes the profile
in question. This matches the production behavior of how sparse profiles are
loaded anyway.
Reviewed By: StanislavGlebik
Differential Revision: D35592415
fbshipit-source-id: 71c1df08ce62ed2aba198940e7cb396e7cc5c96d
Summary:
When processing v2 profiles, the root config (i.e. .hg/sparse) is
special in that any v2 profile it loads will be unioned together. To do this, we
need to mark the root profile as isroot. Unfortunately, any command that
manually loaded a sparse profile directly, bypassed the isroot setting. This
meant loading v2 profiles was incorrect for debug commands.
Let's always set isroot for initially loaded configs. This means we should never
call getsparsepatterns with a sparse profile directly. Instead we should call it
with a .hg/sparse equivalent (like a string "%include $sparse_profile"). The
next diff will update some callsites to do this.
Reviewed By: StanislavGlebik
Differential Revision: D35594323
fbshipit-source-id: 8755d92b4fba49a0a27ae1e6298102f5b33b02be
Summary: Chunk by public node types param has only a subset of `NodeType` as possible values. This diff declares `ChunkByPublicArg` enum to represent these variants.
Reviewed By: markbt
Differential Revision: D35532343
fbshipit-source-id: 9177a1767840c50a32a9f8387ad1ff54dc522b6f
Summary: Interned type params allow to pass `all` as one of the possible values. This diff declares `InternedTypeArg` that represent enum with all the possible values.
Reviewed By: markbt
Differential Revision: D35503706
fbshipit-source-id: 29e51d2601dd0306ef6456d88b611e11f656a9df
Summary: Declare `HashValidationArg` that represents specific possible values for the hash validation params.
Reviewed By: yancouto
Differential Revision: D35503707
fbshipit-source-id: da6890ca0905eae810b97827e47d26db5cfe8b1c
Summary:
The journal serialization format is newline delimited, and if there is
any corruption in the file (like after a reboot, or if multiple processes try to
write at once), the newline delimits could get out of whack. Let's not crash if
that happens.
Differential Revision: D35596518
fbshipit-source-id: b995f57f764ea2c3b31f309424726f1580542184
Summary: Upcoming work needs to return errors that look like `InodeError` today, but are rooted in a path, not an inode. Move most of the functionality of `InodeError` into a new `PathErrorBase` class, and then rework `InodeError` to inherit from that. Also add the new `PathError` for use later.
Reviewed By: xavierd
Differential Revision: D35517826
fbshipit-source-id: 679505e4fd9b1d1548f7bb2d7a2f6dab0b12c6ab
Summary:
hg uncommit would do serial fetches from the network for every file
being uncommitted. In a large commit this could take several minutes. Let's
instead batch fetch the needed history ahead of time.
Reviewed By: quark-zju
Differential Revision: D35589155
fbshipit-source-id: ab37431b70cc8b37615ba93693f22259a0fced78
Summary:
Now when parsing we look for a magic comment e.g. "# source = hgrc.dynamic" that lets you specify additional source information for the (lexically) following rules. This is a hook meant for the rules that get inserterted dynamically from the config.
This approach has two advantages:
1. By embedding the info in the profile text itself it keeps the "fetch the data" callback interface clean and simple.
2. By using a comment, the profile contents will be backwards compatible with the Python sparse handling should it ever get persisted to a file.
Reviewed By: quark-zju
Differential Revision: D35157968
fbshipit-source-id: d69b6505a54fc3948b4733167c84291d6236b7e7
Summary:
Add sparse::Matcher::explain() which tells you what rule if any matched a given path.
To do this we pipe along the rule origins into the Matcher object. It looks up the proper origin based on which rule in which matcher matched.
Reviewed By: quark-zju
Differential Revision: D35157967
fbshipit-source-id: bc1fd46b90ae03ea03bc197152aec4fbc34ce57e
Summary:
Smash all the sparse rules together to create a pathmatcher::Matcher object. The main complexity comes from the v1 vs v2 differences in semantics.
I return a concrete type sparse::Matcher that manually composes multiple TreeMatchers together. I need to keep the objects separate for upcoming "explain" functionality to be able to work backwards from the matcher to the original sparse rule.
Reviewed By: quark-zju
Differential Revision: D35157975
fbshipit-source-id: 2aaad3701c02e3e28924676452016037657174c3
Summary: Move the unioning logic to associated functions. This makes it easy for other matchers to "manually" perform a union operation while still keeping the underlying matchers separate.
Reviewed By: quark-zju
Differential Revision: D35157971
fbshipit-source-id: 34ab9ce970b3bc235aadbf4d285e84ecb0e173bb
Summary: This is a subset of the various prep work Python does. We only support "glob" and "path" rules for now.
Reviewed By: quark-zju
Differential Revision: D35157969
fbshipit-source-id: e6d6dc58fb142a3ef26486a2cf070ac3a3df3d3c
Summary: Fix tests to expect the right path separator. The relativizer stuff says it produces strings to show the user, so I presume showing Windows users backslashes is appropriate.
Reviewed By: quark-zju
Differential Revision: D35478523
fbshipit-source-id: 4cd5ef5e1875b6a02fa19b66061d6d8936e5bd06
Summary:
Rust lacks a lexical-only path simplifier in the std library, but we have it implemented mostly as part of utils::path::absolute(). Split it out in to a separate function.
I introduced some edge case handling to be consistent with Python's os.path.normpath and Go's filepath.Clean.
Reviewed By: quark-zju
Differential Revision: D35157972
fbshipit-source-id: 9a3415fd12e3fe1d7c06ac7d66f22b280ece4a5b
Summary:
Add rules() method to resolve %include statements and return a flat list of a profile's rules. Each rule is also accompanied by a String describing where it came from (i.e. the path of profiles).
I made this async and abstracted the content fetching so non-hg code can use this to parse sparse profile.
Reviewed By: quark-zju
Differential Revision: D35157970
fbshipit-source-id: 3a6d2a20690f57ed6ed38355a3000f174d69036a
Summary: Start of a rust library for dealing with sparse profiles. In this commit we just know how to parse a single profile given the bytes.
Reviewed By: quark-zju
Differential Revision: D35157974
fbshipit-source-id: af4dae36fef08b69e827c6cb4e571d89b7c7ffbd
Summary: Do the same trick we for some other sparse commands to allow usage in eden repos.
Reviewed By: StanislavGlebik
Differential Revision: D35506423
fbshipit-source-id: b02f8f76f18acbbd24431da0d41b8322b9556658
Summary: Use the new and improved hgcmd which supports native rust commands.
Reviewed By: quark-zju
Differential Revision: D35566830
fbshipit-source-id: d9d0fd7fe9bf1cebbeb565341937de2aa9fc5f18
Summary:
This integrates the code added on D35426790 (a65b6b597e) and actually uses it for batch derivation on both deleted manifest types.
This involved doing some logic very similar to `derive_fsnode_in_batch` (https://fburl.com/code/y887jht1), which apparently was done separately for every manifest type. It has a few differences, like calculating the input to the parallel derive separately, and accepting file/dir conflicts.
I also added a tunable so that quick reverts to this change can be done.
Reviewed By: farnz
Differential Revision: D35549630
fbshipit-source-id: a1179878e8a9de48c46703cc18f4a9703592dbb0
Summary:
Right now we continue silently if parent upload to commit cloud fails, which may cause more cryptic errors in Mononoke requests. Let's fail early with a good error message.
I also changed slightly how to find the parents, using the same logic as we use to send the parents to Mononoke, for sanity.
Reviewed By: mitrandir77
Differential Revision: D35499845
fbshipit-source-id: c97359ebd879554b7ac659db194af22689cba55c
Summary: A recent change (D34839333 (acb03a4ce9)) started loading dynamic config for commands run outside a repo. We use dirs::cache_dir() as the config cache dir, but that has proven unreliable in certain production environments. Now instead we also try HG_CONFIG_CACHE_DIR if it is set, and finally fall back to std::env::temp_dir() if all else fails.
Reviewed By: quark-zju
Differential Revision: D35519337
fbshipit-source-id: e24b2a20ee20f358012b3b2326c7537ca206382f
Summary:
I'd like to use debugsparsematch in
[validate_sparse_profiles.py](https://www.internalfb.com/code/fbsource/tools/scm/sparse/arvr/.castle/validate_sparse_profiles.py)
script. The reason for doing that is because it has it's own sparse matcher
logic, which does not support sparse v2, which in turn causes failures on diffs
like D35461391.
Instead of reimplementing sparse match logic I'd suggest to just shell out to
hg (FWIW, we are already shelling out for buck in this script).
My biggest concern is that `debugsparsematch` is a debug command, and so in
theory it's format and/or input arguments can be changed and break
validate_sparse_profiles script. To prevent that I've added a small comment
Differential Revision: D35552480
fbshipit-source-id: cb781d749a7bff2eb8e6d4fcd234962eccc16970
Summary:
The revision numbers aren't stable in this test - rev 6 and 7 might be swapped.
Avoid testing revision numbers and check the graph in a way that works for both
cases.
Reviewed By: sggutier
Differential Revision: D35559269
fbshipit-source-id: 17546c50b0ae9c520a0c59f1f678b91cae81e89b
Summary:
In order to properly support reset on Windows, EdenFS needs to keep track of
both the checked out revision, and the reset one. Since this state needs to
persist across EdenFS restarts, we'll need to store it in the SNAPSHOT file.
This diff merely adds support for this new SNAPSHOT format, but doesn't write
it.
Reviewed By: chadaustin
Differential Revision: D35448641
fbshipit-source-id: 59fbafdadc885dfec8efd119c4fe1ff582987af0
Summary:
This resolves "bug: level n segments [.., ..] are not sorted or connected!" in some cases.
A more ideal solution might be just filling the gaps. For now let's make it stop crashing
first.
Differential Revision: D35548449
fbshipit-source-id: fd79fe2ade866f3ba1501c196d07f75d2b1c11ee
Summary:
This helps detect issues. Changed problems from `assert_eq!` to `assert!` so
they cannot be auto updated by `cargo-fixeq`.
Differential Revision: D35548448
fbshipit-source-id: 84ef027bdff12ee9b73f7dd277f426f861007b4c
Summary:
With discontinuous segments, segments in a group no longer need to be adjacent
to each other.
The `ONLY_HEAD` flag is an optimization. It does not have to be set.
Differential Revision: D35548451
fbshipit-source-id: 3c5396e5429876293fe452507d380a900127e7a9
Summary:
The `BYPASS_READONLY` pushvar can be passed to ignore the lock status
of a repo, or push to a read only repo. This functionality was used by commit
cloud in the early days of the Mononoke rollout. However, we now only check
whether a repo is read only during bookmark movement.
Update Mononoke to check that a user is allowed to use the pushvar before
bypassing it.
Reviewed By: markbt
Differential Revision: D34390881
fbshipit-source-id: e016fbb6ef74de6bc7e4cb92069b984a863da15f
Summary:
By adding some more requirements to RootDeletedManifestIdCommon, it was possible to remove some more duplicated code between DM versions.
I'm doing this to simplify code, and because I'll add the batch derivation on this level as well.
Reviewed By: kris1319
Differential Revision: D35463189
fbshipit-source-id: 9f896e4faf795a00334db773288894a0f110c8fd
Summary:
This is basically a port of D30989888 (35a7998c07) to deleted manifests, and enables batch derivation for them. It does not have the limitation that commits can't delete files and turn them into directories.
The basic idea here is to parallelise the derivation in the tree, not in the commits. I added a comment explaining this idea in code, see the diff's contents.
This diff adds just a helper function for that, which I will integrate into the usual batch derivation trait in a following diff.
I didn't port the optimisations in the non-batch code: Batching some writes to blobstore and using a local mutexed hashmap to avoid double writes. I will add them in a following diff if unless I disprove their efficiency.
Reviewed By: farnz
Differential Revision: D35426790
fbshipit-source-id: 7432617737323a3e3854f4ba347bc86e8043ab2e
Summary:
Previously "hg purge" would do "hg purge --files --dirs". However, that is slow since there is no fast way to find all empty directories to satisfy the "--dirs". Also, this behavior was somewhat surprising because the docs suggest --dirs is not included by default. Change the behavior to default to just --files (but add a config knob so we can revert this behavior quickly if needed).
Empty directories are normally invisible, and my presumption is very rarely the user cares about empty dirs when purging.
Reviewed By: yancouto
Differential Revision: D35515752
fbshipit-source-id: 3d4ab97309de1a9ef82987486fbbd1dbea3b91d8
Summary: markbt pointed out that a couple of items in the API were non-idiomatic. Change them to match idiom.
Reviewed By: markbt
Differential Revision: D35504087
fbshipit-source-id: 2eba98e6be834ae4d44a0d4923a58007750a8139
Summary: I misunderstood the use of MPath in mutable renames. Fix behaviour to match the reality
Reviewed By: markbt
Differential Revision: D35478916
fbshipit-source-id: 90984a28548226e18dd88468de5d41531fa0241d
Summary: Move the `admin hg-sync-bundle` command into `mononoke-admin`. It controls all aspects of `hg-sync` (not just bundles), so just call it `hg-sync`.
Reviewed By: kris1319
Differential Revision: D35395745
fbshipit-source-id: 0fe0a139c0e44ffb84e16ef3154f4cd2f9c0c53a
Summary:
Migrate the specialist `write_stub_log_entry` test tool into a subcommand of `mononoke_testtool`.
The existing tool is a bit of a misnomer: while it's purpose is to add entries into the bookmark update log, it actually does so no by writing stub entries, but by actually moving the bookmark. So let's name the subcommand what it is: `modify-bookmark`.
Reviewed By: kris1319
Differential Revision: D35395744
fbshipit-source-id: 3aff8acfd65090559ca90a4e3f89f2806586a5f9
Summary:
Fix crash on uncommit about missing "renamed" method.
For now we just treat that no renames are tracked. Later we might want to
properly fix rename tracking.
Differential Revision: D35518154
fbshipit-source-id: 4082428d408385f9a2fe22d5720aa81a1b59b5e4
Summary:
The fileset DSL scans through all files. For simple fileset expressions such as
`set:*.py` or `set:a.py or b.py`, convert them to non-fileset patterns for better performance.
Differential Revision: D35490968
fbshipit-source-id: 5452d325040859d02bd66256571803c8229a8b88
Summary:
Change the runtime of the fileset DSL to not eagerly list files if possible.
This should speed up patterns (stringset), and certain filesets like "unknown()".
This should make `hg debugfileset PATTERN` fast.
Differential Revision: D35500777
fbshipit-source-id: d1477de1703ea611aeaa893883c92d474c066cc0
Summary: Change `x in list` to `x in set` to improve time complexity.
Differential Revision: D35500778
fbshipit-source-id: 05b81180b19e171f241483b939e408e4e477525c
Summary: This makes the test pass with `run-tests.py --chg`.
Differential Revision: D35490967
fbshipit-source-id: bbc729103c26565aa7447743ba181642818d1253
Summary: This adds a test for preventing ourselves from adding revsets without docstrings (which would not show up when running `hg help revsets`), and also adds a way of marking revsets that we explicitly want to hide.
Differential Revision: D35372411
fbshipit-source-id: 1b63402fc63bc6cb786b87f14145f653f22f21ce
Summary: Memcache doesn't like spaces in keygens; paths can have spaces in. Use the path hash instead, which is guaranteed to not have spaces
Reviewed By: markbt
Differential Revision: D35478914
fbshipit-source-id: 54fb025bf45361d5ed8ed71d0303c25e1889cf8c
Summary: This hasn't mattered so far, because ChangesetId is usually unique across repos, but this is not guaranteed (e.g. in megarepo merges)
Reviewed By: markbt
Differential Revision: D35478915
fbshipit-source-id: 573797d438db27e0b6897f894602048a5c2b4c27
Summary: I didn't understand this on first run through this code, so made mistakes. Add comments to fix this up
Reviewed By: markbt
Differential Revision: D35478917
fbshipit-source-id: 046caf4dffc9a775795b0e74795383078c714e10
Summary:
This is the same as `eden trace fs`, redirect people to that. The command will
simply be removed later.
Reviewed By: fanzeyi
Differential Revision: D35483104
fbshipit-source-id: bdacda6c5e6f9d8d809f44b64b1178aacbee4228
Summary:
The graph output is slightly changed, probably caused by recent client-side dag change.
It does not affect correctness.
This is found by https://www.internalfb.com/diff/D35478351?dst_version_fbid=693252335199916.
Differential Revision: D35506537
fbshipit-source-id: 35f6cbac28b5ec8205e2cae73040422cf868e880
Summary: Adds a description for `eden trace hg` which outlines the meaning of the different emojis used
Reviewed By: kmancini
Differential Revision: D35473126
fbshipit-source-id: 32c67aa728cf2065255cfbb0227eb52f74d672df
Summary: Similarly to D35294917 and D35256084, this makes the Rust repo object hold the changelog (`dag_commits`) object. Notice that unlike its Python version, caching is not done on disk.
Differential Revision: D35360056
fbshipit-source-id: 04e51dc75665dcc52aa28b09909f5a60b12493b4
Summary:
Adds a method for making the Rust repo object hold the EdenAPI, and uses it for replacing `edenapi.getclient`.
Since most of the config checks done on the Python side don't seem to be useful in the new Rust world (most of them were used for testing), porting them or adding them to the bindings was simply not done.
Differential Revision: D35295652
fbshipit-source-id: 3058b02f2bb18d5a9b0697b0114e2c966fef5b73
Summary: Before this change, the inner Rust repo would load its own config object, which would cause manually configured options to be disregarded. These are used, among other things, to configure paths for loading the EdenAPI.
Differential Revision: D35294917
fbshipit-source-id: a85613b488f7da82590275ea6f755f1b98aad297
Summary: Adds a couple of methods for making the Rust repo object handle the metalog, and makes use of this on the Python side. Since the way the metalog is used in Python is tightly coupled with the VFS object, trying to make use direct of it would be rather complicated, so we are still going to use it through the VFS object.
Reviewed By: quark-zju
Differential Revision: D35256084
fbshipit-source-id: 273ad118ebbeb72e04f43020e2c48cab4958d162
Summary:
Recently, a check was added for avoiding loading the inner Rust repo if `init.use-rust` was not enabled. This was a mitigation for users seeing excessive reloads in the reponame file caused by the reload of the dynamic config, which was caused by the loading of the inner Rust repo object.
This also should be fine since we are rolling that config to users this week anyways.
Reviewed By: quark-zju
Differential Revision: D35300521
fbshipit-source-id: bba2e37217f8bd3cdd11c6091c404389ec3c3b2b
Summary:
This adds some QoL stuff to some traits/structs, like requiring Debug/Clone. Most of this is not
strictly necessary but makes it easier to add debug statements and inspect the code.
Reviewed By: markbt
Differential Revision: D35434704
fbshipit-source-id: bd69ecd06e9186b47a8088ef533d3f7d8d846107
Summary:
On this part of the algorithm, we do a bunch of serial awaits, but they don't need to be serial, we can run them together.
---
Partly related, I am also considering changing `lookup` function to take a vec of things to lookup so we can look them up together more efficiently, which would help here (specially on DMv2), but it might be a bit overkill as cache should already help a lot here.
Reviewed By: markbt
Differential Revision: D35399538
fbshipit-source-id: 19902c2881edb2e078a390972b7e1f7fe3298d86
Summary: This makes the `backfill_derived_data validate` command check for consistency of DMv2 with DMv1, to make sure they work exactly the same.
Reviewed By: markbt
Differential Revision: D35394945
fbshipit-source-id: e92511123665b6d89ceafd473593c1961da8e759
Summary:
This makes it easier to incrementally dump changesets to a file.
I didn't find a perfect way to limit the output, but this should do for now.
Reviewed By: kris1319
Differential Revision: D35359394
fbshipit-source-id: f354f62b34aec4e393c9496af7d4bc2558ad7676
Summary: Functionally the same as the support for DMv1. Needed to add a bunch of nodes and edges everywhere, but luckily the compiler was able to guide me.
Reviewed By: kris1319
Differential Revision: D35355036
fbshipit-source-id: e8fd7a7d39ae92bd2b9173defe6e1296c68c79b6
Summary:
This diff implements all the necessary traits to transform dm2 into "proper derived data". Unfortunately I couldn't break it out further as they all depend on each other.
- BonsaiDerivable knows how to "derive a deleted manifest v2" (which reuses the same code from v1), and to store mapping.
- RootDeletedManifestIdCommon ties the classes together and allows us to reuse the same code as v1.
- This also adds the thrift config for the derived data, and necessary code on test repo to be able to derive it in tests.
Reviewed By: kris1319
Differential Revision: D35319579
fbshipit-source-id: e4670fa2dc2f73ad8f476d51571e25e55169fd0b
Summary:
This diff implements the DeletedManifestCommon trait for Deleted Manifest v2, meaning it can "behave like a deleted manifest".
The trait was added several diffs ago to maximise reuse of code between both manifest versions.
Reviewed By: kris1319
Differential Revision: D35318240
fbshipit-source-id: 19899fb9e02a76feee321898b353417453f1e07f
Summary:
This adds the rust boilerplate counterpart to DeletedManifestV2 added on the previous diff to thrift config.
It doesn't have anything special, only the reading/writing to thrift and to the blobstore.
Reviewed By: kris1319
Differential Revision: D35315948
fbshipit-source-id: c66c4300be0e3bc1f07e06344a62263bca1574d2
Summary: Basically identical to the DM v1 definition, but it uses a `ShardedMapNode` instead of a `map<MPathElement, DeletedManifestId>`
Reviewed By: kris1319
Differential Revision: D35315612
fbshipit-source-id: dcb7963bbcec7dab1f2154b326bf4a1839449582
Summary:
This changes sharded maps so the size is stored on the edge, not on the node itself. See [1] for more discussion.
This has a few advantages:
- No need to load the node to know the size.
- This makes it possible to do some optimisations, see [2]
- Since we can easily look at the sizes of children, we don't need to be super extra careful to keep the correct size while updating the map, and can just calculate it at the end. See [3] where code is a little cleaner.
And disadvantages:
- Code is a little more boilerplaty, as it needs yet another struct to hold the additional size.
- To calculate the size of the current node, we need to traverse the children. That should not be a big deal as we have to load them anyway. I also put it behind a OnceCell so it's not inefficient.
IMO I'm fine with going either way. I think size on edge is a little uglier to code but a little more practical, so I prefer it. But I'm fine to go with size-on-node if someone else has a stronger opinion.
markbt has thought about this and might have opinions. I don't think using a separate node is good as this is only for size (which is necessary for the algorithm to be fast, not "aggregated data").
Reviewed By: markbt
Differential Revision: D35314803
fbshipit-source-id: 13f488dbbc5cf07a82fac9d0e638682c7d9fd926
Summary:
**Context:** Sharded maps will be used on deleted manifest v2
This diff adds the implementation for the `update` function, which generalises all write operations (adding and removing elements). This is the meat of all the "improved deleted manifest" stuff, most of the other things on top will just be adaptors and boilerplate code.
As it generalises both operations, it needs to deal with all the details of keeping the optimisations in place, plus we also need to be careful so that the structure is history-independent (which should be the case if we keep use the optimisations correctly).
This is a complicated diff, and I recommend also reading [this explanation](https://fburl.com/lnusbzgl) to understand the code. I'm also happy to set a 1:1 and whiteboard the logic to make it clearer.
Reviewed By: markbt
Differential Revision: D35288510
fbshipit-source-id: fc2c1eb0f92dff7c131c4e742007886b0d01196a
Summary:
**Context:** Sharded maps will be used on deleted manifest v2
This diff adds Default implementations for sharded maps and its substructs, and also a `store` implementation that stores the current map in the blobstore.
These will be used on update but are reasonably separate from it, so I'm putting it on its own diff.
Reviewed By: kris1319
Differential Revision: D35253317
fbshipit-source-id: 1d71917dff0079608d4a086d11396a1bfb8537d6
Summary:
**Context:** Sharded maps will be used on deleted manifest v2
This diff adds the logic for the `into_entries` operation on sharded maps. That's the equivalent to `.into_iter` on a regular map.
We take special care to only load things in memory as needed, which should mean we never need to load more than a couple of blobs in memory at the same time.
Reviewed By: kris1319
Differential Revision: D35251932
fbshipit-source-id: 0005b34f254180afd78d112a34b2de067f968e80
Summary: I did submit another fix to upstream, then I backported it on top of 0.1.11.
Differential Revision: D35494891
fbshipit-source-id: 4ba29ba35e556926b30e9c47569381d506d1b8a7
Summary: 0.3.10 adds a new EnvFilter builder that I would like to use. :)
Reviewed By: wqfish
Differential Revision: D35473723
fbshipit-source-id: df31dad09c8f8fc9c27a43c85b5d6ac8f97d69a3
Summary:
We have seen that sometimes the a client sends us nfsv4 requests with the
nfsv3 version on it.
We fail to parse those.
It's likely the kernel that is sending us that event, but technically any client
could connect to us, so we don't know that for sure.
Let's make sure we log if we see more than one client connected to us. so that
we can confirm if this is the kernel or not.
We should probably also not let non kernel clients connect to us. But I have not
figured out how to do that. tbd
Reviewed By: fanzeyi
Differential Revision: D35448531
fbshipit-source-id: e0810c8961c18b305b80bb874ae4f6aee9583d07
Summary:
ASIC homedirs are stored on NFS. Eden/hg's new caching solution by default places contents in the home directory which can lead to concurrency issues when reading/writing to this shared NFS cache dir, especially for CI (which all runs as the same user).
To mitigate, we can set XDG_CACHE_HOME to point to a non-NFS directory where cached contents should be written. However, Eden only propagates these allowlisted environment variables throughout the lifetime of the EdenFS daemon. Let's ensure we can also pass XDG_CACHE_HOME through to the daemon process as well.
Reviewed By: chadaustin, mrkmndz
Differential Revision: D35402429
fbshipit-source-id: 0ce5b71838bcf384ea20427cf01b2fe9a107f69c
Summary:
Until now we had 3 different ways of permission checking in Mononoke across our
3 surfaces (scs server, writeproto and eden api). Let's start unifying them by
moving the wireproto logic to from repo_listener to its own facet.
In this diff I'm moving the mononoke-server logic to the new facet as-is (that
is without changing the logic. In the next diffs I will migrate other monoonke
ACL checks to use it.
Reviewed By: HarveyHunt
Differential Revision: D35178977
fbshipit-source-id: befbcb5e78482efc09558838846cc20c76655ae7
Summary:
EdenFS does support answering a status query between 2 commits, by using it, we
can avoid forking to Mercurial which is a very expensive operation. We can also
benefit from EdenFS computing these concurrently instead of running these
status sequentially.
The code is almost identical to the one present in scm/Mercurial.cpp with some
copy/pasted bits. The main difference being the lack of LRU caches. I've
decided against these LRU caches as I've noticed Watchman consumming a large
amount of memory due to the status call potentially containing a very large
amount of strings. Since the cost of asking EdenFS for status should be
significantly less than Mercurial (no Python startup time, no Python code,
etc), not having caches is likely OK.
Lastly, we still fallback to the `hg status` path when alwaysIncludeDirectories
is set to avoid complexifying the code.
Reviewed By: chadaustin
Differential Revision: D35445851
fbshipit-source-id: ae939b71e8f6c60e5724cde5fc42753fddae2570
Summary: Add Rust entry point for clone command. Start by falling back to python. Later add rust clone impl and config for running rust or python clone impl. The flags and doc are copied directly from python https://fburl.com/code/24mjlvza. They'll need to be updated (the docstring in particular is very outdated), once we finalize supported options for the new clone. https://fb.quip.com/sNsnAbxMHD9 (ef5a29b32a)a
Differential Revision: D35232312
fbshipit-source-id: 2c67419308b7f6143f06adc24ecf75e1c54aba65
Summary: We were limiting the minidump uploads to 977KB. This allows the whole minidump file to be uploaded instead.
Reviewed By: fanzeyi
Differential Revision: D35449385
fbshipit-source-id: f4d8966fee5eda312639adbc476bf9f72c1aec43
Summary: By default, we collect every minidump from the last week. This can be a lot. Let's limit the collection amount to 3.
Reviewed By: fanzeyi
Differential Revision: D35449132
fbshipit-source-id: 158f22b6d4083958b26f34bdfc56ba3099aa0be6
Summary: reuse the string from earlier in the function
Reviewed By: kmancini
Differential Revision: D35334023
fbshipit-source-id: 61fc2989d5ca56eaa9c703a1725106ebb60f295f
Summary: Add the ability to enable/disable notifications from the E-Menu
Reviewed By: chadaustin
Differential Revision: D35269912
fbshipit-source-id: d51e224ea4fbf7c4c700573f7f672641a3961883
Summary: the logic used before was wrong. Fix it so that we handle all cases correctly (before we ignored the config option to turn off notifications but handled notification frequency correctly)
Reviewed By: chadaustin
Differential Revision: D35270735
fbshipit-source-id: d0be2e202fb37062e2ed73c909b9bf0e8251b343
Summary: adds a general purpose notification method (send a notification with any title/body combo)
Reviewed By: chadaustin
Differential Revision: D34453071
fbshipit-source-id: 886fcca310c82ca50340570ecebf7182e9a72f7d
Summary:
An old development GUID was used. This caused some users to be unable to start Eden with the E-Menu enabled because the GUID was associated with another executable file path. See here: https://docs.microsoft.com/en-us/windows/win32/api/shellapi/ns-shellapi-notifyicondataa
"The binary file that contains the icon was moved. The path of the binary file is included in the registration of the icon's GUID and cannot be changed. Settings associated with the icon are preserved through an upgrade only if the file path and GUID are unchanged. If the path must be changed, the application should remove any GUID information that was added when the existing icon was registered. Once that information is removed, you can move the binary file to a new location and reregister it with a new GUID. Any settings associated with the original GUID registration will be lost.
This also occurs in the case of a side-by-side installation. When dealing with a side-by-side installation, new versions of the application should update the GUID of the binary file."
Reviewed By: chadaustin
Differential Revision: D35302987
fbshipit-source-id: 8c898a93047ae30b97853ca31400dd1c927a39f4
Summary:
Shell_NotifyIcon(NIM_ADD) fails if there's a stale icon in the notification tray. This caused Eden to crash upon startup.
Instead of blindly adding each time, We should first call Shell_NotifyIcon(NIM_DELETE) to ensure any stale icons are removed.
We should also avoid crashing EdenFS if NIM_ADD fails.
Reviewed By: chadaustin
Differential Revision: D35302853
fbshipit-source-id: cf3f3062f96eada83fcde54616b2407e13779b0c
Summary: The Cmake build didn't properly link the E-Menu resource file. That caused EdenFS to crash on startup if users had the E-Menu enabled.
Reviewed By: chadaustin
Differential Revision: D35288550
fbshipit-source-id: 27f2f29a53ab2717fd9fd874a2cd89b81ccbb9b7
Summary:
"false" shell command in the test sometimes finished too quickly and hit the `hg peer has died` code path so I replaced it with `sleep 0.1`.
I also noticed that for the same reason `ensure_alive_dead_process` test could hit two different code paths so I split this test in two and made them explicitly check both paths.
Testx link: https://www.internalfb.com/intern/test/562950011553744
{F718584633}
Reviewed By: markbt
Differential Revision: D35398464
fbshipit-source-id: c88debf04f5f179e580f51da0ef17db0d96770f7
Summary: This was set to default to true in `EdenConfig.h` but was not set here
Reviewed By: chadaustin
Differential Revision: D35440139
fbshipit-source-id: cf3102bea40ae639c767dd6488924fd843556d48
Summary:
One of the issue some users have seen on Windows on a regular basis is when the
content of files on disk differs from what EdenFS thinks the content should be.
This leads to `hg status` not reporting the correct output and sometimes random
build failures. At first, let's try to detect these occurences. This diff focus
on only computing the sha1 of files that have been fully read already as
reading other files would force EdenFS to populate them on disk.
Reviewed By: kmancini
Differential Revision: D34776469
fbshipit-source-id: d04d31f7f7bfba85cbfbf68ce55f5490bbaec45f
Summary:
On Windows, we've had cases where EdenFS disagrees with the state of the
filesystem. EdenFS may believe that some files are materialized, while they
aren't present on disk. This checker should ensure that we detect these cases.
In a future diff, automatic remediation will be added.
Reviewed By: kmancini
Differential Revision: D34775506
fbshipit-source-id: 3eaa1761fd8acd0fbbfd9f965b35c712c28f2f97
Summary: This could be potentially useful for D35302414 (2e968d8b23).
Differential Revision: D35302972
fbshipit-source-id: b8032a7128839864f1d4f07e7100f18deced4cb3
Summary: Migrate the subcommand and two integration tests that check the correctness.
Reviewed By: markbt
Differential Revision: D35367702
fbshipit-source-id: b908fe7b4404fe52191f6079d96d8378e347089e
Summary: The diff migrates walker corpus subcommand over to the new clap and its integration test.
Reviewed By: markbt
Differential Revision: D35366521
fbshipit-source-id: 73c020377129772eb9fbd079fed9504bed74360e
Summary: Unfortunately the initial test [`test-walker-count-public-chunked.t`](https://fburl.com/test/026x4co5) is flaky, so for this one I also received conflicting signal: 1 out of 5 runs I did failed. The run that failed actually was executed with other tests.
Reviewed By: markbt
Differential Revision: D35364752
fbshipit-source-id: 3a89999932d3446cb03e5bcde2fd094b720aa976
Summary:
Migrate validate command to the new clap,
This diffs also introduces a new enum `CheckTypeArg` that helps to use clap to specify default value for the `include-check-type` arg and resolves directly into `CheckType`.
Reviewed By: markbt
Differential Revision: D35360875
fbshipit-source-id: 21222cd2dbfbc144efa3627af29111d8dd9ecd1c
Summary:
**Context:** Sharded maps will be used on deleted manifest v2
This diff adds the logic for the `lookup` operation on sharded maps. That's the equivalent to `.get` on a regular map.
Reviewed By: markbt
Differential Revision: D35248969
fbshipit-source-id: 465f0a6d02d43647bc225ae318e9ce77613089eb
Summary:
This diff adds the example map seen on the design quip as a test for sharded maps.
It also adds some helper functions to make creation of test maps simpler.
Reviewed By: markbt
Differential Revision: D35248489
fbshipit-source-id: d3d9ee139c9057646f4b598307c8fd3e898c68b2
Summary: This moves v1 specific stuff to `_v1` files in `derived_data/deleted_files_manifest`. Due to all the previous refactoring, this is now just the root manifest id part, as ops and derivation are agnostic to dm version.
Reviewed By: markbt
Differential Revision: D35246790
fbshipit-source-id: 269d2a1f8eb121368b5479899875bfbf40b635c6
Summary:
**Context**: Sharded maps are going to be used in DMv2 implementation.
This diff adds the basic boilerplate code for working with sharded maps in Rust. Algorithms are coming later.
It contains:
- Logic for converting to/from thrift for the Rust sharded map counterpart.
- `is_empty` and `size` methods, which are trivial.
One thing to notice: In Rust, the type is `ShardedMapNode<Value>`, where Value is any type that can be converted to/from binary. In thrift, there's no generics, so we store all values as binary. This is why D35217399 was needed, since we needed to decouple `blobstore_key` from `MononokeId`, as we can't implement that trait for `ShardedMapNodeId`.
Reviewed By: kris1319
Differential Revision: D35219036
fbshipit-source-id: ba4b137c4b60532a18b05ccc4385ff59362b50a8
Summary:
**Context**: Sharded maps are going to be used in DMv2 implementation.
Finally, after a huge stack of refactorings, something new!
This adds the thrift definitions for the sharded map discussed in [this design doc](https://fb.quip.com/ktMoAYnNwbUu#temp:C:ZHDd32bef5d5067b46d1b28a1bc7), with very few differences:
- I renamed "normal node" to "intermediate node" because I think it's clearer.
- I added farnz's suggested way of inlining nodes.
- I'm using binary as the value of the map, as there are no generics in thrift (unfortunately). When parsing this in Rust, we will make sure they match the correct type.
I will soon produce a more up-to-date design doc, taking into account the modifications since the initial one, which should make following algorithm diffs easier to review and serve as a wiki.
Reviewed By: farnz
Differential Revision: D34822820
fbshipit-source-id: 2524d38270c2fbb6db321788d6146d53e9a6a6e8
Summary:
This is a different approach to D34790637.
The issue is that we can't implement `MononokeId` for some ids to come (ShardedMapNodeId), but we want to call the `blobstore_key` method still. When refactoring, I noticed `megarepo_api` already has a neat trait to do this: `BlobstoreKeyWrapper`.
So this diff implements `BlobstoreKeyWrapper` for all ids, and makes MononokeId depend on that instead and not implement `blobstore_key` manually.
Hopefully this approach is less complicated.
Reviewed By: markbt
Differential Revision: D35217399
fbshipit-source-id: 740bbdfaaf68b7e72c1a13987520ff2404d81035
Summary: changes `ImportPriorityKind` and members in `ObjectFetchContext` to `uint8_t`, these should be minimized for use in `HgImportTraceEvent`
Reviewed By: xavierd
Differential Revision: D35269554
fbshipit-source-id: f36752b41e653338704316a8a75e2bbc72317e4a
Summary:
Pulling in this patch until the **[patch](f4f439ca0c)** ~~is approved, lands, and~~ makes it into crates.io
NOTE: I feel a little awkward patching ***everyone*** here — I suspect I'm the only that cares.
**[I attempted to add a "tar-symlink" package pinned to 4.38](https://www.internalfb.com/intern/paste/P491303978/?view=diff)**, with the patch applying to that,
but and then updating only `dotsync2` accordingly, but that results in the same cascading changes
seen here; I suspect due to the patch applying to package "tar" (since "tar-symlink" isn't a thing).
Reviewed By: zertosh
Differential Revision: D35350696
fbshipit-source-id: a5653f3aed0eb5bc5424906d24a53ec2c2b79211
Summary: If one ran `hg help revset`, the `lost()` revset would not appear in the help since it was missing a docstring. This diff fixes that.
Differential Revision: D35363029
fbshipit-source-id: 81c2251d1ec2c6c5397de8d41443141271e8c265
Summary: The diff migrates one of the tests and checks new walker scrub.
Reviewed By: mitrandir77
Differential Revision: D35343497
fbshipit-source-id: 680d6720eaab13e660bee4356f68ba2126679b0c
Summary:
The diff migrates one of the tests and checks new walker scrub.
Some of the arguments now belong to the walker command and not to the subcommand, for example, the scuba log table.
Reviewed By: mitrandir77
Differential Revision: D35285714
fbshipit-source-id: e7e6fca0fe10e063edcb61ff6be88064143cceac
Summary: The diff migrates one of the tests and checks new walker scrub.
Reviewed By: mitrandir77
Differential Revision: D35285713
fbshipit-source-id: 2f6599439a00c87cdf93c72df969d9eb029b4963
Summary: I used the subcommand API from mononoke_app, which removes unnecessary boiler plate.
Reviewed By: mitrandir77
Differential Revision: D35247783
fbshipit-source-id: 58a4204d22a8d68fb636f588b843982bd8affbe6
Summary:
This is a refactoring/reimplementation of the [`setup_common(..)`](https://www.internalfb.com/code/fbsource/[e22d12de8703]/fbcode/eden/mononoke/walker/src/commands/setup.rs?lines=1308) function and which is heavily based on the old clap and cmdlib infrastructure. The function is huge and hard to read, I tried to split new one into the smaller functional pieces.
For some reason in the old setup the repos are initialized sequentially, I preserved behaviour in the new one, but intend to make it parallel in a separate diff.
Reviewed By: mitrandir77
Differential Revision: D35247782
fbshipit-source-id: aa5cd4ee2f221f62f988765445ae29d8c7289b45
Summary: This diff sets up common arguments for the different walker subcommands. The diff doesn't add any new functionality.
Reviewed By: mitrandir77
Differential Revision: D35247784
fbshipit-source-id: 8a75b53a335a62a1e30983ccbbed0c7cdeef952b
Summary:
`ScrubOptions` has default for `queue_peek_bound` which got overwritten by `environment_hook()` in scrubbing (new cmdlib). So if the peek bound wasn't specified in scrubbing, the default in `ScrubOptions` will be reset to `None`. I found it while working on D35343497.
Looking at how this field is used, I think there is no reason for it to be an optional value. It has a max default, which was introduced here D28533393 (fdee7b86e9). So this diff makes the field non-optional and fixes scrubbing `environment_hook()` so it won't override options to `None`.
Reviewed By: mitrandir77
Differential Revision: D35330717
fbshipit-source-id: 32a87dbb75a5b0cbe593d50427d4c36509a74b61
Summary: Update `mutable_counters` to futures 0.3, and convert it to a repo attribute using the `facets` library.
Reviewed By: Croohand
Differential Revision: D35323567
fbshipit-source-id: 217f1e28c9541404a98a54b48222265efaba52ad
Summary: We shell out to "hg debugnetworkdoctor" because it is a native rust command.
Reviewed By: quark-zju
Differential Revision: D35302414
fbshipit-source-id: 0806f66d1694e845eeacfc6b4535f0d29a529b8a
Summary: Improve the message the users sees when the network doctor is run automatically due to a fatal network error. We now clearly state the command failed due to a network error, and we write the detailed output to a file. Previously we just showed the network doctor output and it wasn't clear the command had failed.
Reviewed By: quark-zju
Differential Revision: D35255833
fbshipit-source-id: 764abce8dcb7796283abc4b9fe1a485e04ff74c7
Summary:
Stack context:
To get a build setup for edencommon I picked an eden library to move over
to edencommon.
I picked process name cache. I would like to have this library in watchman
so that I can log spawning process command lines as well as client command
lines.
This diff:
finally actually moves over process name cache
Reviewed By: chadaustin
Differential Revision: D34218020
fbshipit-source-id: 1e51ec6524d1e67a1dfd4d03a8189f882f5d3ff1
Summary:
Stack context:
To get a build setup for edencommon I picked an eden library to move over
to edencommon.
I picked process name cache. I would like to have this library in watchman
so that I can log spawning process command lines as well as client command
lines.
This diff:
process name cache has some benchmark tests so I am moving benchharness over.
Reviewed By: chadaustin
Differential Revision: D34218024
fbshipit-source-id: 260730adb7e5d4a3131e72fa2f03cc4c5066c075
Summary:
Stack context:
To get a build setup for edencommon I picked an eden library to move over
to edencommon.
I picked process name cache. I would like to have this library in watchman
so that I can log spawning process command lines as well as client command
lines.
This diff:
process name cache depends on StringConv.h so I am moving that over.
Reviewed By: chadaustin
Differential Revision: D34218023
fbshipit-source-id: d928768a6b5f737a1c546c6f3c73acc5535fc44c
Summary:
Stack context:
To get a build setup for edencommon I picked an eden library to move over
to edencommon.
I picked process name cache. I would like to have this library in watchman
so that I can log spawning process command lines as well as client command
lines.
This Diff:
process name cache depends on WinError.h so I am moving that over.
Reviewed By: chadaustin
Differential Revision: D34218026
fbshipit-source-id: 064f5bfc50c4ba120c887b01ff1f0a3e0543d498
