Summary:
Now that `deep_sharding` is expressed via `ShardingModeConfig`, the repo-level `deep_sharded` boolean flag can be removed.
Corresponding configerator diff: D42676137
Reviewed By: markbt
Differential Revision: D42676143
fbshipit-source-id: 09b87aab1718f828c3015930645ba58ca58bb1fd
Summary: Replace `BlobRepo::name` with the method on the facet trait.
Reviewed By: yancouto
Differential Revision: D42682284
fbshipit-source-id: 40fbcfb10fe3fc9e96f4ad6afd504a93184af9a5
Summary: Adds an integration test for the features added in D42221073
Reviewed By: bolinfest
Differential Revision: D42625035
fbshipit-source-id: 40a00f36ce25bd2250637bca3b42f16076be1304
Summary:
D42614252 (79886f53d4) introduced a way of setting `default.default-push` through the `SL_TEST_GH_URL` environment variable in tests. This diff makes use of the same environment variable for `github.submit.get_origin`.
This can be useful when testing commands that need to connect to the mock GitHub server but still need to get data from the git repo being tested.
Reviewed By: bolinfest
Differential Revision: D42625037
fbshipit-source-id: bf19d772fef32bf4ce8326391a9df360a3412040
Summary:
This allows revsets of the form `PR[0-9]+` to be translated to commit hashes as well as autopulling them in case a local copy doesn't exist. This revset can be used in commands that support revsets, such as:
- `goto`
- `log`
- `show`
- `rebase`
- `pull`
Marking the translated revsets as remote bookmarks is still pending
Reviewed By: bolinfest
Differential Revision: D42221073
fbshipit-source-id: 695fe82d0f0c23ea32e18c86be537ed3ea0807f7
Summary:
https://doc.rust-lang.org/nightly/nightly-rustc/rustc_lint/for_loops_over_fallibles/static.FOR_LOOPS_OVER_FALLIBLES.html
Both Option and Result implement IntoIterator trait, which allows using them in a for loop. for loop over Option or Result will iterate either 0 (if the value is None/Err(_)) or 1 time (if the value is Some(_)/Ok(_)). This is not very useful and is more clearly expressed via `if let`.
Reviewed By: muirdm
Differential Revision: D42656184
fbshipit-source-id: b14b53b49ef90eea26a2329dbe37168e5d454dea
Summary: removed unnecessary 'hg' command in the test and fixed some typos.
Reviewed By: muirdm
Differential Revision: D42656185
fbshipit-source-id: 4300b32703230193df425193715bfd0974872695
Summary:
Detect cargo features and run with features turned off. This can help capture
issues turning default features off.
Reviewed By: muirdm
Differential Revision: D42642444
fbshipit-source-id: 4a65d71a1e4e9766f385456c6738e2a01f644aca
Summary:
This avoids invalid feature selections such as `--no-default-features` when
testing `cpython-ext`.
Reviewed By: muirdm
Differential Revision: D42643921
fbshipit-source-id: 517a47c81aed49ca8ebf0c7b8f4bdcb9eba8bba0
Summary: It is only used by the website SVG renderer. Let's make it optional.
Reviewed By: muirdm
Differential Revision: D42590209
fbshipit-source-id: 17231e8468792dfb87cc2010d4b74257fb5257b6
Summary:
First, don't use "--keep" since that results in a duplicate set of commits visible in smartlog. It also makes it hard to map between the rebased commits since it doesn't record mutation markers. Further, note that "--keep" does not output the "nodechanges" data, so we were always assuming the rebase did no work, and would proceed to push the un-rebased commits. This would lead to a "non-fast forward" error whenever the default branch was newer than the stack's base.
Second, use "rebase -r" instead of "rebase -s". "-s" was rebasing the entire stack in the first loop iteration which causes two problems: 1) It rebases too much (i.e. it can rebase later commits in your stack you aren't trying to land right now). This is bad because you could see conflicts in commits you aren't trying to land, which doesn't make sense. And 2) it messes up the loop's mapping of commits since the loop assumes it is rebasing one commit a time. This issue was probably moot due to the "--keep" issue.
I also added some logic to "goto" back to the un-rebased commit after the rebase. In general, landing commits should not change the working copy. We also "hide" the commits created by rebase so the user won't see duplicate commits in their smartlog if there is a failure.
Finally, I tweaked the rebase logic to be more conservative wrt commits being "rebased out". If one of the commits being landed is already on the target branch, we abort and ask the user to rebase manually.
Fixes#333.
Reviewed By: bolinfest
Differential Revision: D42203379
fbshipit-source-id: 39a8dcd4d65e0535fc2379e4ad16fea902bb88d0
Summary:
Add a test that demonstrates an issue w/ "ghstack land" logic. The main issue is the pull/rebase never affects what we end up pushing, so non-fast-forward errors will occur whenever the remote main branch has commits newer than the stack's branching point.
I added/tweaked the mock utils based on what queries ghstack submit/land uses. I added a more generic "expect_request" which callers can use to mock out a response without needing to create a specific type.
Reviewed By: bolinfest
Differential Revision: D42614252
fbshipit-source-id: b9928361128f22c2f456836075aa7cf7ead5be11
Summary: Some users of make_request import the function, so subsequent wrapping of make_request has no effect. Now the wrapping applies to the unexported _make_request and always take effect.
Reviewed By: bolinfest
Differential Revision: D42614253
fbshipit-source-id: bc7c7b74333984c54adfc3d49a6207c46ee2f9b6
Summary:
I mainly want this so the HGUSER env var is respected in tests.
I prefer ui.username()'s abort message to what get_identity_or_raise() had since it cuts to the chase instead of "forcing" the user to open the link to discover the recommended command for setting username.
Reviewed By: bolinfest
Differential Revision: D42614255
fbshipit-source-id: de4a6719bfdefb2298e316bd18bad2c564aded84
Summary: Instead of randomly picking repo from available for worker. Let's make a 1:1 mapping, so each worker thread will be responsible for single repo. And we can control how many repos per tasks will be using SM. I also made the derived_data_config to be default - which is the one used by our prod jobs. Backfilling will be added later as a separate set of derivation worker
Reviewed By: RajivTS
Differential Revision: D42649996
fbshipit-source-id: fb3f31876472c232f5422a07480465f91af052ec
Summary:
We're working on removing `BlobRepo` functions to make it easier to deblobrepoify targets.
This removes `get_repoid`, which is used in many places, but it's just an alias to the `id()` function in `RepoIdentity` repo attribute.
Reviewed By: markbt
Differential Revision: D42675456
fbshipit-source-id: ef4c884f92ce0dcc769db6b105c5819863b19492
Summary:
I already made `add_recursive` more efficient on D42605099 by supporting calling with multiple input commits.
However, we still do many serial calls to `CommitGraphStorage::add` inside it, which can be slow. This diff starts using a `BufferedCommitGraphStorage` on top of our commit graph storage, which does all the serial things in-memory, and then writes them in parallel to the underlying storage, which is much faster!
Reviewed By: YousefSalama
Differential Revision: D42610183
fbshipit-source-id: 2c731e49b272cb5ff7fba85ba0954a16f4ea42a3
Summary:
This splits the traits and useful types to a separate `commit_graph_types` library.
This makes things clearer, but also, will allow me to depend on some `CommitGraphStorage` concrete implementation from within the `CommitGraph` structure on the next diff.
Reviewed By: YousefSalama
Differential Revision: D42609914
fbshipit-source-id: cf72a429a1db0db2bd27217bbed94447a1e0e435
Summary:
This makes Changesets::add_many more efficient in the compat struct (assuming the inner implementation is also efficient) by:
- Call the inner `add_many` implementation
- Modify `CommitGraph::add_recursive` so it can add multiple commits at the same time.
- There was an inneficiency where it could visit nodes more than once (because of the order of checking `contains_key`). It has been fixed as well.
- Do old and new changeset in paralllel. No reason not to.
Reviewed By: YousefSalama
Differential Revision: D42605099
fbshipit-source-id: a3fb5f99668345c816bec66298255e4bbcc671f3
Summary:
This creates a method to do batch additions to changesets. This will be used to optimise commit mirroring.
For now, I provide the dummy implementations for it: just do everything in order.
The optimised implementations will come in the following diffs.
Reviewed By: YousefSalama
Differential Revision: D42547866
fbshipit-source-id: 5366ab95c30d495efd8445fb48e2daf927e27f16
Summary:
This diff does the necessary work to enable writing commits in parallel to the blobstore on the mirror_commits binary.
It does so by cheating slightly:
- To create the commits, we need to look up their parents, which usually means things can't be done in parallel. But since we're doing bytewise commit copy here, we point to the source repo for parents, since it already has all commits.
- SQL information can't be done fully in paralllel, because it uses some autoincrement indexes that can't be copied from the source repo. So I split this up from `CreateChangeset`, and do it manually later, still serially, though the next obvious optimisation is to batch that.
Other smaller changes in this diff:
- Remove `--derive-all` option. This made it trickier to do things in parallel, and deriving data here is not the most efficient way.
Reviewed By: RajivTS
Differential Revision: D42550984
fbshipit-source-id: 4f0ef70821e3ffed31730f3d43c4c71ab15e6bfb
Summary: query HEAD name, instead of assuming main/master
Test Plan:
---
Stack created with [Sapling](https://sapling-scm.com). Best reviewed with [ReviewStack](https://reviewstack.dev/facebook/sapling/pull/386).
* __->__ https://github.com/facebook/sapling/issues/386
Pull Request resolved: https://github.com/facebook/sapling/pull/386
Fixes https://github.com/facebook/sapling/issues/375
Ran a number of affected integration tests:
```
./run-tests.py test-git.t test-git-submodule.t test-git-submodule-rebase.t test-rust-clone.t test-doctest.py
```
Using the example from https://github.com/facebook/sapling/issues/375,
cloned the repo and verified that it cloned the `trunk` branch:
```
$ lhg clone --git https://github.com/cli/cli
$ cd cli
$ lhg
@ 179e9c256 Yesterday at 14:13 2159081+qoega remote/trunk
│ Add projectsV2 support to issue create, issue edit, pr create, and pr edit (#6735)
│
o 06ae07f97 Yesterday at 07:25 mislav
├─╮ Merge pull request #6880 from cli/setdefault-bare-repo
│ │
o │ 351226d34 Yesterday at 07:17 mislav
├───╮ Merge pull request #6881 from cli/reviewers-json-fix
│ │ │
o │ │ 241f9197e Wednesday at 21:15 rmw
├─────╮ Merge pull request #6815 from cli/intake-doc
...
```
Reviewed By: quark-zju
Differential Revision: D42597162
Pulled By: bolinfest
fbshipit-source-id: 2f4b3869f452b57290f2a7a69721ecdd4e444b2e
Summary: Currently, 'hg sl' template is calling 'shelved()' revset function to add '(shelved)' label, and it crashes if there are '._*' binary files created by MacOS. This diff ignores the '._*' files.
Reviewed By: quark-zju
Differential Revision: D42647182
fbshipit-source-id: 9ff8b67e85c55bf2ea24af774f1b638798e71f4c
Summary:
Currently, `hg sl` template is calling `shelved()` revset function to add '(shelved)' label, and it crashes if there are `._*` binary files created by MacOS.
This diff is to add a test to repro the bug
Reviewed By: muirdm
Differential Revision: D42647185
fbshipit-source-id: 72b1677cc2cdc9c084b9e2a5a778d27d4a6c0abc
Summary:
The feature delegates to dag-types/for-tests. Crates using this feature can
enable it on dag-types instead.
Reviewed By: muirdm
Differential Revision: D42591209
fbshipit-source-id: aa0d5c788377cc7500fd0fc10dedf90d2d8d0583
Summary:
Partially reverts D34191759 (0cc5984cde). `quickcheck_arbitrary_derive` is not published on
`crates.io` and is blocking `dag-types` from being able to be published.
Reviewed By: muirdm
Differential Revision: D42591210
fbshipit-source-id: f8d92af0d142bde15900757c9bb58fd19813ad1d
Summary:
This partially reverts D22970570 (c448e0f575). There is no circular dependency because a crate and its
test are treated differently as 2 crates. So `dag`, `dag (test)`, `renderdag`,
`renderdag (test)` are 4 crates. A dependency graph like:
dag(test)
+--renderdag
+--dag
renderdag(test)
+--dag
+--renderdag
is not circular and can be handled by cargo just fine.
Reviewed By: muirdm
Differential Revision: D42590207
fbshipit-source-id: 25e917f78f088be13711d652adbec3e5d6e47e4b
Summary:
It's only used by Mononoke caching and not used by other users.
See https://github.com/martinvonz/jj/issues/1056 for context.
Reviewed By: muirdm
Differential Revision: D42590206
fbshipit-source-id: 6fa25844775a91b9e4aabb23c2c499d4e4570e70
Summary:
This copies the new `fuse_rename_in` fields from the final public release of osxfuse's `fuse_kernel.h` into our macOS FUSE header file, so that our header is now identical to the public release (aside from fixing a shift overflow issue).
This breaks compatibility with old versions of osxfuse prior to the ABI change.
With the header updated to match macFUSE's ABI, we can also remove the runtime check for an extended `fuse_rename_in` in FuseChannel.cpp.
Reviewed By: chadaustin
Differential Revision: D42617239
fbshipit-source-id: 2cf3aaec87488f53dc5f3fec439b7c34c2e5a6b5
Summary:
macFUSE (previously osxfuse) started as an open source project based on the Linux FUSE protocol. It has since gone closed source and diverged from Linux's FUSE. One area of divergence has been in the set of capability flags that the kernel extension advertises to filesystem implementations in its INIT message.
In D14506747 (1a1dbccb1d) we imported a header file from the then-open source osxfuse as a source of truth for protocol messages and capability flags. In D22744378 (8d56e5ccbc), when osxfuse was still open source, we added to this definitions for `FUSE_NO_OPEN_SUPPORT` and `FUSE_NO_OPENDIR_SUPPORT`, corresponding to the capability flags that exist in Linux FUSE. However, these flags never made it into upstream osxfuse or macFUSE.
Somewhere between versions 4.2.0 and 4.4.1, macFUSE enabled a new capability flag with the value 1<<24, which happens to collide with our internal-only `FUSE_NO_OPENDIR_SUPPORT`. Seeing this bit set, Eden incorrectly believes it can take advantage of the `NO_OPENDIR` optimization by responding to opendir requests with ENOSYS. But because macFUSE doesn't support this behavior, listing directories would fail:
```
% ls fbsource/fbcode
ls: fbsource/fbcode: Function not implemented
```
This diff makes Eden compatible with macFUSE 4.4.1 by re-importing the final open source version of osxfuse's `fuse_kernel.h`, which undoes the definition of `FUSE_NO_OPENDIR_SUPPORT` in D22744378 (8d56e5ccbc), with two differences:
- The additional fields in `fuse_rename_in` are left for a child diff, as adding these will break compatibility with old osxfuse.
- `FUSE_XTIMES` is modified to avoid a shift sign overflow.
Reviewed By: chadaustin
Differential Revision: D42594733
fbshipit-source-id: b56fb8a6ad7e2269ef9d1d397f3d7b90821e229a
Summary:
In D38786500, I introduced `deep-sharded` flag that controls whether a repo needs to execute in deep-sharded mode or shallow-sharded mode. However, the flag was universal (i.e. it was applicable to all services). Since we currently have a situation where certain services need to execute repos in deep sharded (e.g. `SCS`) while others need to execute in shallow-sharded mode (e.g. `Eden API`). This diff introduces `ShardingModeConfig` which determines if a repo is deep sharded for a particular service.
For now, the `deep_sharded` flag is kept as is. Once the config + code changes to use this new property are landed, I will delete the old `deep_sharded` flag.
Corresponding configerator diff: D4250164
Reviewed By: mitrandir77
Differential Revision: D42531443
fbshipit-source-id: 697496cc8351f0d9e686312fd8fff8d4e2fa010b
Summary:
In D38786500, I introduced `deep-sharded` flag that controls whether a repo needs to execute in deep-sharded mode or shallow-sharded mode. However, the flag was universal (i.e. it was applicable to all services). Since we currently have a situation where certain services need to execute repos in deep sharded (e.g. `SCS`) while others need to execute in shallow-sharded mode (e.g. `Eden API`). This diff introduces `ShardingModeConfig` which determines if a repo is deep sharded for a particular service.
For now, the `deep_sharded` flag is kept as is. Once the config + code changes to use this new property are landed, I will delete the old `deep_sharded` flag.
Corresponding configerator diff: D42501648
Differential Revision: D42530372
fbshipit-source-id: 1bd46533ad18939da4465239dab1340a0e8219d2
Summary: This makes the test a bit stronger.
Reviewed By: zzl0
Differential Revision: D42493269
fbshipit-source-id: d51ef1f6c0e9c5d7d72dd2e27c1800f0e62c16e9
Summary:
Use locking internally to avoid racy concurrent writes that could cause
unrecoverable data corruption.
Previously:
Timeline: -------------------------->
Process 1: open(), append, append, ...
Process 2: open(), append, append, ...
Now:
Process 1: open(), lock { append, append }, ...
Process 2: open(), lock { append, append } ...
`append()` to the same file is exclusive to one instance.
Note we need to update `position` after obtaining the lock to pick changes from
other instances.
Reviewed By: zzl0
Differential Revision: D42492667
fbshipit-source-id: 6a6184c945d23e16eb1619458abdb18afc64868c
Summary: It'll be used by the next change.
Reviewed By: zzl0
Differential Revision: D42492670
fbshipit-source-id: 676320fc4fd14411a6feca450b53a0b67e82f6ec
Summary:
It'll be used by the next change. Note `Rc<RefCell>` or `Arc<RefCell>` won't
compile for the rest of the project.
Reviewed By: zzl0
Differential Revision: D42492672
fbshipit-source-id: 28b167b1d1e61a715f86911ea1d314fc2420eafd
Summary:
trait FileReadWrite is used for both on-disk and in-memory files. Add locking
so the next change can use it to avoid races on the physical filesystem.
Reviewed By: zzl0
Differential Revision: D42492668
fbshipit-source-id: 06c02ea4fb52213c3a64fc17a336a5edba385dc4
Summary: Currently, 'hg rebase' is rebasing hidden commits and making them visible to users, this is a regression caused by D42087621 (96b767efde).
Reviewed By: quark-zju
Differential Revision: D42608733
fbshipit-source-id: de4b7b231d42a45844174e4a417d3c07a6869b97
Summary:
Now that we have a notion of version, we should log it during startup. This makes our logs easier to read through.
I think including the platform name would also be worth doing.
A little bit unfortunately, we have the platform name in the client platform, but we really want to log it on the server. So here I also just duplicate the platform name into the server platform definition.
We only have 3 implementations right now:
- browser
- vscode
- android studio
Reviewed By: bolinfest
Differential Revision: D42589867
fbshipit-source-id: be2f3783b4d471fa9d503a9ff3d4464f7e5954ab
Summary:
We need in our analytics&bug reporting to know the version number of a given instance of ISL.
The idea here is that we'll have a version number set on startup, which we'll keep constant. This was not a given before this stack!
(There may be situations where reloading an `sl web` page could load newer client javascript from disk than the server side was started with—we'll need to address this in the future for the assumption of a constant version number to really fit)
Version numbers depend on the platform/embedding we're working with:
- `sl web` is basically just the `sl` version, since it's built and deployed alongside `sl`
- for the VS Code extension, we want to use the vscode extension version (from the package.json), because it's updated separately from sl itself.
- Both of these may differ for internal versions vs external versions. Though analytics only happen on internal builds, so it shouldn't matter much (should be clear from context if internal/OSS).
Note that these version numbers may be different from the version number of `sl` that we shell out to, even for `sl web`. Imagine spawning `sl web`, then some time passes, and the `sl` binary is updated on disk. Running `sl version` would give a different version than what ISL calls its version. But that's OK, since the API layer between ISL and `sl` is just command running, and we will need to be weary of backwards compatibility there at all times—we don't assume these versions match.
To start with, we won't log the actual `sl` version for commands we spawn in our analytics. We'll correlate ISL analytics with our other `sl` analytics instead, which should give that info.
Reviewed By: bolinfest
Differential Revision: D42589825
fbshipit-source-id: e51c9eac84a9825720885654854a2d773aa03a19
Summary: Add arguments to isl.py so that each ISL server is spawned with --slversion as expected
Reviewed By: bolinfest
Differential Revision: D42563631
fbshipit-source-id: 9a493c4ddf3f7529f032bc130cbd164fb9d4a668
