Summary:
## What
Expose an argument for a timestamp that sets the lower bound of the commits. It's just exposing the one from `ChangesetPathHistoryOptions` so that we're able to export history only up to a specific date.
Differential Revision: D49230397
fbshipit-source-id: dad7b78d6a9b88072c13cd9c409fd5e0b9d56c92
Summary:
## What
Expose an argument to specify which changeset to start the history search from. We currently use the master bookmark, which will remain the default behaviour if one isn't provided.
I'm also adding some `info` logs that will be helpful to track the progress of the tool.
## RFC
I had to add the `BonsaiSvnrevMapping` trait to the `Repo` definition of MononokeAPI in order to use the `parse_commit` function in the CLI (which I extracted in D48896898).
Can that be a problem? None of the tests failed, so there shouldn't be any impact on business logic, but just wondering if there are other consequences that I'm not aware of... Asking because I imagine that the `Repo` in Mononoke API is a dependency of many other things.
Differential Revision: D48954151
fbshipit-source-id: c3a815bccd3d8372f2ff0caacd7b06e9f8c9456f
Summary: Build and install the isl-dist.tar.xz so `hg isl` can use it on *nix platforms.
Reviewed By: sggutier
Differential Revision: D49560037
fbshipit-source-id: 3664707e371cd58b97f4c702c6cc36c1642e5aa9
Summary:
Update the pick-python logic so it:
- Prefer `PYTHON_SYS_EXECUTABLE` if set.
- Support missing `CFLAGS` in sysconfig. It's not set on Windows.
Reviewed By: sggutier
Differential Revision: D49560365
fbshipit-source-id: 96327157593a65bbc2c2b0475ebceff11aff4bce
Summary:
Let `make local`, `make oss` that use `setup.py` generate the `isl-dist.tar.xz`
in the `setup.py` directory.
Reviewed By: sggutier
Differential Revision: D49558559
fbshipit-source-id: 813f4843adfaef95d3b76b141c86f8991e647a50
Summary:
Change ISL logic from using dotslash and (OSS) Javascript source tree to the
`isl-dist.tar.xz`. This is one step towards unifying OSS, internal non-buck,
and buck builds.
The locations of `node` and `isl-dist.tar.xz` are configurable.
Reviewed By: sggutier
Differential Revision: D49558560
fbshipit-source-id: 6f19f66ae31081ca1cfc79290c2cfe219b47ebe4
Summary: The data local or cache directories are handy in the Python world too.
Reviewed By: sggutier
Differential Revision: D49558557
fbshipit-source-id: 5f57a4e89ba88a68cc0b91c332fed57b74788956
Summary:
This script takes the ISL source, uses `yarn` to build, then write the built
result to a tarball. It maintains a "source_hash" and skips building if the
destination "source_hash" does not change.
The yarn build commands and directories to include are from `isl/release.js`.
Reviewed By: evangrayk
Differential Revision: D49558562
fbshipit-source-id: 060fec91a1f65cda51efa525ec291cd8483cf42b
Summary:
Add logging for `hg edendebugimporter`
HgBackingStore will:
- Attempt to get the data from local cache
- Fetch it remotely from Mononoke
- Or, as a last resort, fall back to using `hg edendebugimporter`. This is a path we would like to deprecate and logging accesses will help us understand who is falling back to this.
Reviewed By: jdelliot
Differential Revision: D49424377
fbshipit-source-id: 959c6a42bb5a5c3df6b9bc206527d0ed28253bbe
Summary: During multi-step operations like rebase, we move the working copy to different parents. Previously the Rust workingcopy's "status" calls to EdenFS were not using an up-to-date parent id in the "status" args, resulting in the "requested parent commit is out-of-date" error. Fix by pulling the parent id out of the treestate for every "status" call.
Reviewed By: quark-zju
Differential Revision: D49509139
fbshipit-source-id: 7dff48e826c073897c5fd1f683ac76f2f0068753
Summary: "eden rage" in particular times out regularly at 20s. Let's crank it up since "eden rage" important.
Reviewed By: zzl0
Differential Revision: D49289239
fbshipit-source-id: 7e8b4869f78a171e55665b5b6e550cda1451ccd7
Summary:
as a part of this initiative https://fburl.com/workplace/ki1ksbyc IG Business is exploring possibility of using templates for summary / test plans to empower authors to write better diffs and make them easier to review.
this diff introduces custom default values for those fields and uses gatekeeper-based gating to roll it out for a limited audience.
Reviewed By: evangrayk
Differential Revision: D49484684
fbshipit-source-id: d8544e393d858b07e111ddb6de1b41709d92cdf8
Summary: For initiator services, if `ClientRequestInfo` is missing, we need to generate it. This diff adds support for that. Note that even if the service is not an initiator service (e.g. `LandService`), we populate `ClientRequestInfo` with `ClientEntryPoint` set as `LandService`. Ideally this scenario should never happen since the caller should always be a known entry point. Presence of such entries in the scuba logs will indicate fishy business that we need to investigate and fix.
Reviewed By: liubov-dmitrieva
Differential Revision: D49485744
fbshipit-source-id: f65aa7e9e6462dd7b92f122a6e7125b62d7eb0cf
Summary:
Continuing to migrate the entire `derived-data` subcommand from `mononoke_admin` to `mononoke_newadmin`.
Preserve the core logic from the [old implementation](https://www.internalfb.com/code/fbsource/[69f5866ebb425a0866c0b8832e6f1e21dccd820b]/fbcode/eden/mononoke/cmds/admin/derived_data.rs?lines=405), but move it to `newadmin` where the code can be improved.
The integration test for `newadmin-derived-data` is extended with tests for `count-underived`.
Reviewed By: markbt
Differential Revision: D49277803
fbshipit-source-id: 8c3a64a89562444c550bc422d99ca58089418acf
Summary:
We now have a newadmin implementation of `derived-data exists`.
This is enough functionality to remove the dependency on `mononoke_admin` from the `backfill-derived*` integration tests which still pass after this change.
Reviewed By: markbt
Differential Revision: D49277802
fbshipit-source-id: 4973f5a9e7b0423bb15a2c61694174d786274323
Summary:
This is the first diff towards moving all `derived-data` subcommands to `newadmin`.
The functionality is preserved from the [`mononoke_admin` implementation](https://www.internalfb.com/code/fbsource/[84a43e29bc998caa347be60af17798a60944b73c]/fbcode/eden/mononoke/cmds/admin/derived_data.rs?lines=369), but the structure of the arguments is modernized with recent clap and `mononoke_app` conveniences.
We loose some convenience by switching from the old `resolve_csids` to `ChangesetArgs::resolve_changesets`. Namely, we can still pass multiple changesets or bookmarks thanks to the previous diff, but we can't mix and match styles, such as passing some bookmarks alongside some hg ids.
This is deliberate to keep the code idiomatic while giving enough flexibility for common uses.
We also force the user to be explicit about the separate arguments (as opposed to positional arguments). This is deliberate as it makes for better ergonomics and is consistent with other `newadmin` subcommands.
`mononoke_admin derived-data` didn't have an integration test, but we create one for `mononoke_newadmin derived-data`.
Reviewed By: markbt
Differential Revision: D49271579
fbshipit-source-id: 57ca4263d80a97e5c6c3d81da637937ce8f5d6f3
Summary:
Later in this stack, we port `mononoke_admin derived-data` to `mononoke_newadmin`.
One functionality that doesn't currently exist in the `mononoke_app` framework yet, and that `derived-data` currently relies on is that of passing multiple changesets to a CLI tool.
We judge it would be a UX regression to limit this app to passing a single changeset at a time.
`derived-data` uses `resolve_csids`, which also gives mix-and-match functionality between different ways to describe changesets in the same list.
We deliberately don't support this because it would make the implementation less idiomatic, using old clap constructs to preserve order, and offer a flexibility that is not necessary from a user's perspective.
Reviewed By: markbt
Differential Revision: D49271578
fbshipit-source-id: fd72a885c36bd4c4fc810d821dcd75324d00fe81
Summary:
Log the same fields as Rust dispatch code. Now we will have structured info for all options/args.
The intention is to use a low retention for the positional args and option values since those will have high cardinality, but keep the option names around longer. The option names in particular are useful for checking where/when particular CLI flags are in use (as opposed to trying to regex match against the unstructured "fullcommand").
Reviewed By: zzl0
Differential Revision: D49509123
fbshipit-source-id: d95c1e2cea46534247b6c3a981d7a48a0d7d17c9
Summary: cpython_ext::Str provided compatibility for Python 2 strings, but we don't need that anymore.
Reviewed By: quark-zju
Differential Revision: D49509127
fbshipit-source-id: 64484feb1525fac77ca1013916f1e328f07f592b
Summary: Now we log positional args, option names, and option values to the "sampling" file. The intention is to use a low retention for the positional args and option values since those will have high cardinality, but keep the option names around longer. The option names in particular are useful for checking where/when particular CLI flags are in use (as opposed to trying to regex match against the unstructured "fullcommand").
Reviewed By: quark-zju
Differential Revision: D49509126
fbshipit-source-id: 4f5803a084de74d68df66356fd8eaf11fa3e9dff
Summary: Keep a list of the names of options that were given to the command. This allows you to differentiate options set to their default value.
Reviewed By: quark-zju
Differential Revision: D49509128
fbshipit-source-id: 97326daa2063efd6a8c68c06cbb06b980115856b
Summary: Minor cleanup to get rid if Str vs. OptStr types. This makes things more compatible with Python's `None`.
Reviewed By: quark-zju
Differential Revision: D49509124
fbshipit-source-id: ea7c0d9a5f14cf7efa95ba63a4c9d0c66e89b0a9
Summary: This way any code can skip tracing and directly output a "sampled" value. I want to log a list of values, which isn't convenient using tracing.
Reviewed By: quark-zju
Differential Revision: D49509125
fbshipit-source-id: c6a684dcb7a13572460ddb9a7678ffebee80f9ce
Summary:
We would like to know how effective our various caching entities are. Here we are adding counters in Overlay to track hits and misses when loading an overlay directory.
File operations in the Overaly are only used when the overlay checker is being ran - as such not adding counters for those.
Reviewed By: MichaelCuevas
Differential Revision: D49552392
fbshipit-source-id: 1d2e699b4b9c5a1bad3d47ef1e7945a5f7f37999
Summary: We would like to know how effective our various caching entities are. Here we are adding counters in EdenStats to track lookup hits and misses in InodeMetadataTable.
Reviewed By: MichaelCuevas
Differential Revision: D49548923
fbshipit-source-id: bd48b71213ecb28d6f0316a7d69955b80aefdb37
Summary: To aid in build times, switched to using forward declarations to remove an depedenciny on EdenStats.h in IndoeMap.h.
Reviewed By: chadaustin
Differential Revision: D49546029
fbshipit-source-id: 1306d28465527e7d6109d413f7e9f4ecb2e7e74e
Summary:
This diff adds multiprocess join and multiprocess terminate to lstat call so that eden deadlock won't hang the process
A timeout lstat for fuse will be reported as hanging mount and we can take different action
There is a TODO part about how to deal with hanging mount that is not implemented in this diff
Reviewed By: kmancini
Differential Revision: D49516967
fbshipit-source-id: afa8ec1eeec8069ba6666dac86a648cd2ea16807
Summary: Version bump and changelog entry for new OSS vscode extension release
Reviewed By: quark-zju
Differential Revision: D49552335
fbshipit-source-id: 2adb0b13e847a461b5e1f53355741ef16cad2f8c
Summary:
I realized we would much rather use the real Commit in the unsaved edits confirmation, instead of a bespoke rendering of commit titles.
This gives us:
- remote synced message (though not the locally unsaved edited title I guess)
- proper commit bubble
- diff status, comments, time stamp, build status, etc.
- expected responsive behavior
Note we need to hide a few things from the commit to make this looks more natural, such as the selection background and the unsaved message dot.
Reviewed By: quark-zju
Differential Revision: D49551902
fbshipit-source-id: 8af2da6a767d4811adc85c63c5fd75da8dd2ae00
Summary: Autofocus the button so you can press enter immediately if you know what you're doing
Reviewed By: quark-zju
Differential Revision: D49551517
fbshipit-source-id: e7c3c6fdf15a1237c1898b47141bba3a2f89a19b
Summary:
Finally, we can fix the bug this stack has been about.
When we go to confirm the split/edit stack, we now know the user has confirmed or cleared their unsaved message changes via the modal. We've also now loaded the edited commit message into the edit stack data structure. This means `debugimportstack` has now imported those edited messages.
Now all that's left is that we need to prevent the succession of the stack from resurrecting the edits once again. Since they've been imported now, it's safe to discard all the unsaved edits for the entire stack.
You may think that this is unnecessary because we already have the same message, so it wouldn't detect the message as being different. This is only true for 1:1 predecessor:successor changes and falls apart with non-1:1 edits like split and fold.
The only tiny edge case is that if the split fails, we've now cleared their message without it getting to get saved. I think this should be an unlikely case—debugimportstack is designed not to fail as much as possible (we should instead fail earlier at the debugexportstack stage, which is safe)
Reviewed By: quark-zju
Differential Revision: D49551311
fbshipit-source-id: dadccda9b462a47d9b4bbdd48cdeb519f5df5672
Summary:
Like the previous diff which shows a confirmation before split from right click, we need the same confirmation before *any* edit stack. This confirmation was already set up to handle multiple commits, but needed some tweaks now that it actually does handle it.
Technically, the only dangerous case with edits is from split making weird successors. But we don't know beforehand if you'll split when using edit stack. So we need the entire modal to handle it. I think this is generally more consistent and better anyway, it could impact fold and ensures the UI is OK to show the latest commit titles.
Reviewed By: quark-zju
Differential Revision: D49551316
fbshipit-source-id: b28bc36666c86435bd6453e7b2505737d5102894
Summary:
When we load the commit stack via exportStack, it doesn't account for the latest remote message OR any unsaved local edits.
Due to weird predecessors, we can't easily reassociate edited messages after splitting. So we show a modal to confirm if the user wants to apply their local edits. If they confirm that, then the idea is to apply the local edits when loading the split UI. This allows us to skip running metaedit for every message with changes and then waiting for that to complete.
The end result is the same, because running split will effectively apply those local edits.
Reviewed By: quark-zju
Differential Revision: D49551315
fbshipit-source-id: 22e9f57b69f7897bd0a70ed4368b42045fcd3b3e
Summary:
Add confirmation modal before you open the split UI, to ask if the user wants to confirm their unsaved edits to the commit message.
This is only required because split can produce strange successors, which will later cause ISL to reapply edited commit messages to unrelated commits from the original. This can duplicate messages and diffs and other weird stuff.
The confirmation here lets the user chose to immediately discard or immediately save the results.
This diff adds the discard path, but see later diffs in the stack for the "save". It won't actually save just yet, it just loads the edited message into the split UI's data, so that when you DO split, it will apply the changes. This allows us to skip waiting for metaedit and doing complicated syncing.
The confirmation here also shows you which commit(s) have unsaved changes, and which fields are unsaved, like title/test plan etc. I think this will be helpful to confirm things.
I believe there's a bug (which I haven't tracked down) in message syncing that can make a commit appear edited even if it was just due a remote syncing difference. So showing the user what's changed makes it easier to think about (IMO).
Ideally I guess we'd even show the diff if you hover the fields...but I don't do that just yet.
Reviewed By: quark-zju
Differential Revision: D49551314
fbshipit-source-id: 7581726c8c18e7459319a5be9b6f8bf626169da8
Summary:
When computing hasUnsavedEditedCommitMessage, we have to compute the fields that are being edited for the commit, and then basically check if any are true.
We can make a selector dedicated to the first part, which finds all fields being edited, and use it to compute the second part trivially.
This lets us reuse the fields being edited which is useful.
Reviewed By: quark-zju
Differential Revision: D49551317
fbshipit-source-id: fd945f456e1bb1362bc01e1ef067ff49393b5960
Summary: I want to use this util in more places, so let's move it to a common file
Reviewed By: quark-zju
Differential Revision: D49551313
fbshipit-source-id: 11172f41da7028851b0c87a2837b3e9113dfff6f
Summary:
I was getting annoyed having the create react app error overlay appear for unused variables. It's super common for me to be developing and want to see my code change even if I have an unused variable.
Let's just make this a warning instead of an error
Reviewed By: quark-zju
Differential Revision: D49551312
fbshipit-source-id: 1567098cf3a4cbb65ad8cea8c1b0150e01decbf5
Summary:
Let's construct singular id for a client. This is intended to be equivalent to SR's client_id. We'll use it for dynamic rate limitting - we'll track usage against the main_client_id and dynamically rate limit if it goes over threshold.
For now we're inferring it just out of identities, but later one we might want to add other info like sandcastle alias. The logic of how the main_client_id is constructed is subject to change.
Reviewed By: RajivTS
Differential Revision: D49440821
fbshipit-source-id: 8f18186e28420cfd17b14afc6323440876620aef
Summary:
before, only `C++`, `Cpp` and `cpp` worked for C++ codeblocks. this adds `c++`:
~~~
```c++
~~~
parallel of D49538396
Reviewed By: bolinfest
Differential Revision: D49538279
fbshipit-source-id: 09b97c688ec99f6903b40a894bcc2a820dae851e
Summary:
## Purpose
We want better visibility in EdenFS cache hit metrics.
As part of this effort, we want to create new ODS counters for different parts of the stack, to see which caches get hit.
At the bottom of the stack, edenFS may rely on the revisionstore to actually query Mononoke for blobs, LFS objects or aux data.
## What we do
Before this diff, we already collected information about cache hits, cache misses etc. on fetch operations of the filestore, but we didn't publish them to ODS.
After this diff, we publish these metrics to ODS.
## How we do it
We use `stats::dynamic_singleton_counter` to create ODS counters for each of the `FilestoreFetchMetrics` that we collect.
We use a dynamic counter so we can create the appropriate counters without having to manually list the exploded list of metrics that are already represented by the `FilestoreFetchMetrics` type.
We use a singleton counter so that the aggregation is done consistently on the Rust side of the binary. This is necessary as ODS counters rely on thread local storage to communicate between the aggregation and the stats gathering. That does not work across languages as Rust and C++ may interpret the bytes in thread local storage differently based on how each language encodes their type.
To make the singleton counter approach work, we need a `FacebookInit` object.
We get it with `assume_init`, which is the only way to get hold of the `FacebookInit` that was generated by calling `fbinit` in C++ from Rust. `FacebookInit` is not ffi-safe, so we couldn't pass it down if we wanted to.
To avoid calling `assume_init` in a context where `fbinit` was never called, we feature gate the ODS code behind an `ods` feature.
We create a new library: `revisionstore_with_ods` which uses this feature.
Only `backingstore` now depends on `revisonstore_with_ods`. `backingstore` is the entrypoint between EdenFS and the revisionstore. This means that only code from `EdenFS` which definitely called `fbinit` will exercise the code branch with `assume_init`.
This incidentally allows us to only add the `stats` and `fbinit` dependencies to `revisionstore_ods`, which means we don't affect the `revisionstore` library which is also used by Mercurial itself.
Reviewed By: MichaelCuevas
Differential Revision: D49438188
fbshipit-source-id: c44eaa46deb00f37165ab13e3525c409c09af03e
Summary: This diff updates the implementation of `ClientRequestInfo` to allow for optional `main_id` since the client will not have the `main_id` (due to difficulties in computing it at client-end) and they still need to send the `ClientRequestInfo` header. This diff also adds a `new` method for creating `ClientRequestInfo` with `None` main_id along with a randomly generated `client_correlator`. It also extends the set of possible `entry_points`.
Reviewed By: zzl0
Differential Revision: D49483588
fbshipit-source-id: 4172271d1f4da8b78c8660462c87244a9d24bc62
Summary: InMemory InodeCatalog rebuilds the Overlay from the filesystem only. This results in an InodeMap that does not load Inodes for non-materialized directories. Updating test to reflect this possible, but correct result.
Reviewed By: kmancini
Differential Revision: D49477523
fbshipit-source-id: 0b9038151f368ecf9ac527a1b491e32c3510db18
Summary: The InMemory overlay does not persist any of its state so on restart it can leave the repo state in a weird state. This change allows us to rebuild the overlay without having a root inode.
Reviewed By: kmancini
Differential Revision: D49441198
fbshipit-source-id: b703a95147fff595ed963bbd6de2a2fb835100ff
Summary: InodeMap, in subsequent diffs, will report metrics when looking up inodes (hits, misses, errors). Adding a new stats subtype with counters for this purpose.
Reviewed By: chadaustin
Differential Revision: D49477890
fbshipit-source-id: 18386cd18e83da57053f39f6eea2966de48dbfb5
Summary: InodeMap, in subsequent diffs, will report metrics when looking up inodes (hits, misses, errors). Adding EdenStats to it initalization to enable this.
Reviewed By: chadaustin
Differential Revision: D49477891
fbshipit-source-id: 86f7c357d5cf011897d1ccb57e42f80f55890ba0
Summary:
remove old header to pass over the client correlator
we are migrating to the ClientRequestInfo that will contain client correlator and will be passed between the services as part of ClientInfo in a different header called "X-Client-Info"
Reviewed By: quark-zju
Differential Revision: D49482058
fbshipit-source-id: 56fb5f44c836d14a8be5a9fd1ab94527764eba06
Summary:
take into account RateLimitStatus in load shedding
we shouldn't enforce RateLimitStatus::Tracked and RateLimitStatus::Disabled limits, this is wrong.
the track we will just log (it will be implemented in a separate diff) and disabled we just should ignore
Reviewed By: clara-9
Differential Revision: D49521366
fbshipit-source-id: 8e62271ebe43640349dfa9334c726481c65d7b81
