Summary:
The negotiation part of the git protocol involves the client specifying its `WANTS` and `HAVES` and the server realizing the set of objects to send to satisfy the client's need. The first version of the server expected only commit Object Ids as part of `WANTS` and as a result it failed straight away in the presence of tags. The current version of the server allows for tag Object Ids as part of `WANTS` but it converts it into commits pointed to by the tag. That works in the cases when the commit pointed to by the tag is part of the packfile being sent by the server.
If the packfile does not include the commit that the tag points to, then we completely omit sending the tag even though the client explicitly requested it. This integration test highlights that we do not support this behavior yet and fail with a fatal in such a case. I will fix this in a follow up diff along with an updated integration test.
Reviewed By: mitrandir77
Differential Revision: D56979846
fbshipit-source-id: 4cff34b4d5ba55a652b91d57ce5ac95a1ca0b162
Summary:
All the integration tests related to `git pull` used it as `git pull https://hostname/repos/git/ro/repo.git`. This doesn't function as a standard pull, i.e. just `git pull`. [This SO post](https://stackoverflow.com/questions/48749715/what-does-git-pull-link-to-remote-do) explains the detailed difference but fundamentally using pull with a remote URL ends up using a default ref spec that works only for branches (i.e. `+refs/heads/*:refs/remotes/origin/*`). This makes us miss out tags from the server.
This diff updates the integration tests to make use of just `git pull` so that we pull all refs (heads and tags).
Reviewed By: mitrandir77
Differential Revision: D56979746
fbshipit-source-id: a309ef656dfb1e179cf342041729a9e5920539e6
Summary:
In repositories imported from Git there are two dates: the author date and committer date. Depending on whether they were imported into Mononoke first (which uses the same style as hg-git), or imported into Sapling directly, these dates are mapped differently.
In the cases where Mononoke did the import, we may want to configure repositories to prefer the committer date, rather than the author date, which is what is used by default. Add a new config option `log.use-committer-date` which does exactly this.
We also extend the JSON format for `hg log` to include both author date and committer date if they are available. This means tooling that knows specifically which one it wants can be modified to fetch exactly that.
Reviewed By: quark-zju
Differential Revision: D56941671
fbshipit-source-id: 0d68a333caf06b8b82e08b581f06df6394520992
Summary:
This diff adds a new private enum to differentiate between the different types of Trees that can be get from backing store.
All `getTree()`, `getRootTree()`, and `importManifestForRoot()` at some point call `retryGetTree()` or `importTreeManifest()` to get Tree from backing store. This new enum is passed in the params then these two functions will know which type of Tree they are getting from baacking store. We will use it to collect correct metrics.
## Note
This doc explains all the SaplingBackingStore ODS counters/duration after this stack changes: https://docs.google.com/document/d/1o355e8JGvq3fBYFD738Jkx9tPSWzH4bmxcM8YRKMziU/edit?usp=sharing
Reviewed By: kmancini
Differential Revision: D56842474
fbshipit-source-id: 0699e215ba7d488ac2c2b873fb97ca331dc207f2
Summary:
the diff eliminates dependencies on brotli-3.3.4 and brotli-decompressor-2.3.5.
the reason is for the avoidance of duplicate symbols that result from combining targets that depend on `fbsoure//third-party/rust:brotli_decompression` (e.g. `rerun_c` via `ureq`) AND `fbsource//third-party/brotli:brotli_decode` (e.g. `fbsource//third-party/apache-arrow:arrow`).
with these changes in place then remove occurrences of the link flag `"/FORCE:MULTIPLE"` introduced by D56797954 and D56821592 to work around this problem.
Reviewed By: jdonald
Differential Revision: D56943460
fbshipit-source-id: b38d69503f6be85a2055a89da2a161d114030c4c
Summary:
- This crate provides a parser for PCI configuration space
- Really useful for writing tools for PCI configuration space that don't just shell-out to `setpci`
Differential Revision: D56983535
fbshipit-source-id: 0407d9e0f856a9b7042f82180f32ee4aedd0436e
Summary: I skipped this during the update references impl.
Differential Revision: D56903500
fbshipit-source-id: 12e75b993ca6d40ff5c994125b6bf460b23420e4
Summary:
# Context
Genna discovered that passing junk to `eden debug blob` on a FilteredFS repo would cause the Eden daemon to crash. This is because `XCHECK_EQ()` eventually resolves to `XLOG(FATAL, ...)` if the equality check fails[1]. We don't actually want to crash in this case, since a user can pass non-sense to 'eden debug blob' whenever they'd like
# The Fix
We should just throw an invalid argument exception if the ID we get passed in looks like garbage. This causes the thrift call to fail instead of making the daemon crash.
In addition, I moved the `XCHECK()` lower in the function to be an `XDCHECK()` so that we only crash the daemon in debug mode (not production). This is fine, since that check is meant to be hit during development, not in produciton!
Reviewed By: genevievehelsel
Differential Revision: D56846354
fbshipit-source-id: 3623eb05ccff83b4186b7f4e4363fa769df872bd
Summary: increase the timeout for jest tests in CI. We previously increased the `waitFor` timeout, but we shoudl also bump the general test timeout too.
Reviewed By: quark-zju
Differential Revision: D56954680
fbshipit-source-id: 65145914071d4452f0487e486e593ebf13f75aff
Summary:
Even with the changes from D56443510 it's not enough to make Sapling look for
local Git commits in case there is a newer Phabricator version. Sapling commits
do not have this issue since they are backed up by commitcloud, so those usually
get autopulled if Phabricator's version is more recent than the local one.
This diffs changes that for Git repos, showing a warning telling users to use `jf get` to get the latest version since that error message should be more useful for them.
Reviewed By: muirdm
Differential Revision: D56447240
fbshipit-source-id: 54025ac7f330c629b3b804a3b9dc1a3013e29d83
Summary:
Some time ago Phabricator revsets were usable on Git repos, but seems like
recently something broke and it's not possible to use them for finding
local commits anymore, which this diff fixes. Additionally, this diff
makes it possible again to find public commits by Phabricator revsets,
which seems to have been broken for a much longer time.
There are two root causes for these issues:
- `remotefilelog.reponame` has the name of Git repos as `reponame` instead
of `reponame.git`, which in turn causes `sl` to error out with issues like
`abort: D123 is for repo foo/bar/baz.git', not this repo ('baz')`
(fixed in D56444313)
- The logic for resolving Git revisions from Phabricator was written 10 years
ago, and depends on using sshpeer which is not really supported anymore: D1458491
Reviewed By: muirdm
Differential Revision: D56443510
fbshipit-source-id: 14e176cac5de1216b1e170d0863d7b7c6382df59
Summary: Git repo names are not being properly compared with JSON results from Phabricator, causing various issues as explained in D56443510, which this diff fixes. This is however not enough for making Phabricator revsets work again, but that is done in D56443510
Reviewed By: muirdm
Differential Revision: D56444313
fbshipit-source-id: c646de21bfa5c6d0813f5d66977aa90f9f799908
Summary:
This makes most of our existing tests have the equivalent configs and files in the repo that adding `$ configure modernclient`. There are a few reasons for doing this:
1. This makes eventually migrating tests to be compatible with EdenFS considerably easier, especially if they don't have `configure modernclient` in them
2. This reduces overhead for creating new tests and makes it easier for test writers to use the recommended setup
3. This opens the path for doing arbitrary actions for setting up a test. Before this diff the best we could do was editing configs in `default_hgrc.py`, but this already does some file manipulation for all tests.
4. This allows modernizing only debugruntest tests, since Mononoke tests aren't really compatible with it right now and just enabling more modern files breaks a number of those tests. Also, the option for using more modern configs in our own test without breaking the Mononoke ones possible.
Reviewed By: muirdm
Differential Revision: D56766291
fbshipit-source-id: dfa4a34e9dcb2784610d77cfeaa629bb6dea492d
Summary:
Now, the `SaplingBackingStore` know the source of each request. However, the sourced is available only for the finish event. The start and queue event doesn't know the source. In the last diff we used UNKNOW for these events. This diff changed the FetchedSource to a std::optional. Therefore for these events we don't need to populate it.
On the other hand, eden.thrift has a NOT_AVAILABLE_YET option. It chooses this option when the std::option doesn't have value.
The kFetchedSource map also update then if we have NOT_AVAILABLE_YET as the fetchedSource, it will give us a blank space character " " and the trace hg will print blank space char for start and queue events.
Reviewed By: kmancini
Differential Revision: D56942308
fbshipit-source-id: 732d5b4fd11cd2e9c5b8ff8e6dc1ec70bc262033
Summary:
Revive integration tests. These had broken from a bunch of changes along the way
- fix jest config to match normal jest tests and transforms and version upgrades
- make sure tests don't import from `isl/src`
- use explitily `refresh()` to make it less flakey
- wrap cleanup in `act()` to remove warnings
- bump timeouts
- make logging output be annotated with the source: `[server]`, `[client]`, `[test]` (for messages during test or in test infra), `[c -> s]` and `[c <- s]` (for RPC messages)
- add README to explain how to write integration tests
TODO in another diff: get this working on CI
Reviewed By: zzl0
Differential Revision: D56916269
fbshipit-source-id: 4b2c997a26fb2476ae996859f7205205b769cc1f
Summary: If you click refresh, it should refresh as much data as possible. Somehow that wasn't already including merge conflicts. Make sure clicking refresh rechecks the conflict state. Most of the time, this does the cheap `.sl/merge` file check and doesn't requiring the full `sl resolve --tool internal:dumpjson`.
Reviewed By: zzl0
Differential Revision: D56916270
fbshipit-source-id: b34a472e587873b51b22044b1d0e83489c512c93
Summary:
When dequeuing items from the zelos derivation queue, first we query zelos for all nodes in the ready directory, and if the ready directory is empty, we will end up sleeping for 10 ms (current value of "scm/mononoke:derivation_worker_sleep_duration_ms") and then querying zelos again. For most repos this is excessive and puts a large load on zelos unnecessarily.
This diff instead makes use of Zelos' ability to set watches, by setting a watch on the ready queue when it's empty, and only querying zelos again when the watch returns a notification.
Differential Revision: D56939625
fbshipit-source-id: 6ac6c420b63c793ffd7b94d195f77173c4020a13
Summary:
We'll be needing it soon to support backsyncing and I'm working on fixing a bug that's partly caused by the fact that `CommitSyncRepos::new` takes submodule deps but drops it silently when creating `LargeToSmall`.
The call to `get_submodule_deps` then always returns `NotNeeded`.
This is the case because when I initially implemented this, we were under the assumption that we would only be supporting forward syncing.
Reviewed By: mitrandir77
Differential Revision: D56881703
fbshipit-source-id: bb90fbc39b25d4fadc5342dafa35c02f93b25bd0
Summary:
Small refactoring to simplify the logic in `CommitSyncRepos::new` and also reuse code later.
Also simplified an if-else block in commit_syncer
Reviewed By: mitrandir77
Differential Revision: D56826691
fbshipit-source-id: 5c24538d5c1bc5079d6cd0e88113afb11077e600
Summary:
Just implementing debug for some data types so I can use `dbg!` with these.
Also small formatting changes
Reviewed By: mitrandir77
Differential Revision: D56826269
fbshipit-source-id: 3c6cc44bda6fa3f59a5d441d6bb61a5edeb2b046
Summary:
In D56706360 I started automatically deriving fsnodes, because to expand submodules from a commit, you need the fsnodes from its parents.
But in the merge process, we currently manually backfill derived data after running the initial import.
I don't think this makes much sense, so let's automatically derive everything by default.
One note though: I don't want derivation to block the import's progress, so I'm spawning a tokio task for all the types except fsnodes.
Differential Revision: D56877240
fbshipit-source-id: c0da1998a8b73c75c88c9bc0cd08f7c3730ca3f4
Summary:
I'll try to make these small changes as I work on the codebase.
Forward syncer always uses small repo as source and large repo as target, so naming them properly will make its code easier to understand.
Differential Revision: D56817782
fbshipit-source-id: 9e0cba01755e2b08d1c2b3875711258ba41f63c1
Summary:
This Diff implements the EdenAPI Suffix Query handler.
The handler extracts the changeset from the eden context and uses it to call find_files_with_bssmv3.
The output of that function is a stream of MPaths. These are transformed into SuffixQueryResponses and returned in and async stream.
This Diff also implements several traits for CommitID to help in the functionality of above.
Reviewed By: markbt
Differential Revision: D56433194
fbshipit-source-id: 28085e6c345cb33bfa33f9cef9c9ed5708d483dd
Summary: This Diff implements a stub handler for handling suffix queries on EdenAPI. The stub handler only returns a static message. It will be implemented in a followup diff
Reviewed By: liubov-dmitrieva
Differential Revision: D56275449
fbshipit-source-id: 3234d7d1fe48744bde3e0c4c7eead75cbd482075
Summary:
This PR implements the client and py bindings for EdenAPI suffix queries.
The endpoint will be at /suffix_query
Return types are currently placeholder. They will be updated once the handler is set up
Reviewed By: liubov-dmitrieva
Differential Revision: D56275448
fbshipit-source-id: 68b0dfb9ba6ba4893a95c37b2f13d11ded9ecb34
Summary:
This diff impelemnts the type skeletons that will be used for making suffix queries to EdenAPI.
SuffixQueryRequest should be close to what the final version will look like, but SuffixQueryResponse should change when the handler is actually hooked up to the BSSM querier
Reviewed By: liubov-dmitrieva
Differential Revision: D56219419
fbshipit-source-id: 0d6cc11b0932c32bdfa7f4d1e647cadcaccc4ec0
Summary: This is done in preparation for D56766291
Reviewed By: muirdm
Differential Revision: D56828983
fbshipit-source-id: 182004749ea74b422440c221b8601c5903ad03b1
Summary:
D56592499 accidentially changed the order of things so load_dynamic never got a auth_proxy.unix_socket_path (since `self` is now empty).
Fix by generating and inserting the dynamic config later. This way, auth_proxy.unix_socket_path can be configured anywhere (except for the dynamic config).
Reviewed By: quark-zju, sggutier
Differential Revision: D56942632
fbshipit-source-id: c98e616993b03374667702552d3c07be00ffdb95
Summary: All the directories needed by the zelos derivation queue are setup when constructing the queue, so if the ready directory is missing that should be propagated as an error not just a warning.
Differential Revision: D56939624
fbshipit-source-id: 9aa6ba6541ce121ff1b12ae1071cfd762dfaed32
Summary: Using `whoami::devicename` causes issues on macOS sometimes, plus it gives us funny looking hostnames on POSIX in general, so let's drop it. Using environment variables to getting this value as well as getting the username is probably fine, so let's do that instead.
Reviewed By: mitrandir77
Differential Revision: D56724499
fbshipit-source-id: b36a08a48121cecfdb34ed33f68422d687b0346d
Summary: Don't truncate bookmarks as aggressively, it makes them hard to read
Reviewed By: jakebolam
Differential Revision: D56943465
fbshipit-source-id: e849ba75c0b1d614cf3c063f5238ee613d41715b
Summary: The current state of this code causes an infinite loop if trying to use `getStats` for the `eden stats` cli in D56453268. This diff introduces a specific helper function to extract these singular values. Also, we don't need to collect the whole of the `Stats` objects anyways to feed these two dynamic counters anyways.
Reviewed By: kmancini
Differential Revision: D56728227
fbshipit-source-id: 22523f97a6864464beed875e82fef1d99348fe2a
Summary:
getdeps can generate a windows wrapper script that can be used to run build artifacts from build directory.
In github actions for large projects we set a getdeps option delete the build dir as soon as we've successfully installed artefacts to save disk space. This option was enabled for windows in D56165825. Turns out that didn't work, this diff adds the missing conditional so that it should.
Reviewed By: vitaut
Differential Revision: D56930778
fbshipit-source-id: 0cb9ac94ef9b39f4e33af8fb91098dc0d833731b
Summary:
NOTE: If your `hg bisect` brings you here & the error you are seeing looks like ` expected Result<&PyAny, PyErr>, found Result<Bound<'_, PyAny>, PyErr>` then see these [migration notes](https://pyo3.rs/v0.21.0/migration.html#from-020-to-021) for the fix or click on `fbcode/security/ace/pyo3/authz.rs` or similar files from bellow to see what the fix is!
In order to upgrade `pyo3` to [`0.21.1`](https://github.com/PyO3/pyo3/releases/tag/v0.21.1), the following had to take place:
## [PyO3]
* Address [migration notes](https://pyo3.rs/v0.21.0/migration.html#from-020-to-021) for `Bound<'py, T>`
* Address [#3595](https://github.com/PyO3/pyo3/pull/3595) - this is done in a crude way for now since there are many call sites depending on `fbcode/dba/rust/common/service_address/py/pyo3_conversion_helper.rs` which would require a more thorough review.
* Address [#3821](https://github.com/PyO3/pyo3/pull/3821) - `pyo3-build-config` is now dependent on and used by PyO3 macros. Currently, the only thing that gets checked is `abi3` compatibility. To address that, we introduce a fixup for that automatically generate `pyo3-build-config*.txt` configuration files, given an `fbsource` Python version. We are tryin to stay as close to `pyo3-build-config*.txt` spec as possible even though only a single bool from that file is ever since there is validation for the other fields but also to future proof future changes. By also generating this file ourselves, we prevent PyO3 from attempting to locate a Python interpreter some other way which seems to start leaking into the PyO3 API as an implementation choice already.
## [PyPi + Rust]
* Upgrade `orjson` to [`3.10.1`](https://github.com/ijl/orjson/releases/tag/3.10.1)
* Added `README.md` notes for future upgrades
* Upgrade `py-polars` to [`0.20.22`](https://github.com/pola-rs/polars/releases/tag/py-0.20.22)
* Removed `py-polars` and `polars` from `target_os = "windows"`. The `third-party/pypi/polars` Python extension has only been supported for Mac and Linux for a while now so its only natural to do that on the Rust side as well. What is more, `polars-util` is bringing in `stacker = = "0.1.14"` which does not build on Windows mostly because its using a much more recent version of `libc` than we use in `third-party/rust` (see P1228807344)
* Upgrade `pydantic-core` to [`2.18.2`](https://github.com/pydantic/pydantic-core/releases/tag/v2.18.2)
* Removed old `third-party/pypi/pydantic-core` versions
* Upgrade `safetensors` to [`0.4.3`](https://github.com/huggingface/safetensors/releases/tag/v0.4.3)
* Patch `third-party/pypi/cryptography/41.0.7` to account for PyO3's [#2975](https://github.com/PyO3/pyo3/pull/2975) (`0.19.0`) where `pyo3::once_cell` was renamed to `pyo3::sync` (see D56826865)
* Upgrade `tokenizers` and `tokenizers-python` to [`0.19.1`](https://github.com/huggingface/tokenizers/releases/tag/v0.19.1)
* Removed old `third-party/pypi/tokenizers` versions
* Fixed `third-party/pypi/tokenizers/BUCK`
* Migrated `third-party/pypi/tokenizers/0.19.1/BUCK` to mirror other Python packages that bind to Rust crates e.g. `libcst`, `polars` etc.
* Removed Windows support from `third-party/pypi/tokenizers`
## [Rust]
* Upgrade `indexmap` to [`2.2.6`](https://github.com/indexmap-rs/indexmap/releases/tag/2.2.6)
* Both latest `pydantic-core` and `c2pa` depend on `serde_json > 1.0.112` which brings in `indexmap = 2.2.1`. The latter has deprecated `.take` and `.remove` on both `IndexMap` and `IndexSet` leading to a bunch of errors (see bellow), all addressed:
```bash
error: use of deprecated method `indexmap::set::IndexSet::<T, S>::take`: `take` disrupts the set order -- use `swap_take` or `shift_take` for explicit behavior.
--> fbcode/hphp/hack/src/package/types.rs:76:16
|
76 | self.0.take(value)
| ^^^^
|
= note: `-D deprecated` implied by `-D warnings`
= help: to override `-D warnings` add `#[allow(deprecated)]`
```
Reviewed By: capickett
Differential Revision: D56671179
fbshipit-source-id: 3ae69c069b7f005570c1a06d37194cf056282a18
Summary:
This test was added a few days ago in D56701219 and has been failing ever since.
In [the sandcastle logs](https://www.internalfb.com/sandcastle/workflow/1265511495293864711/actions) for oss-mononoke-linux-getdeps, we can see:
```
failures:
---- test::test_get_stats_no_max_set stdout ----
thread 'test::test_get_stats_no_max_set' panicked at common/memory/src/lib.rs:63:9:
assertion failed: stats.is_err()
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
failures:
test::test_get_stats_no_max_set
test result: FAILED. 2 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
```
Reviewed By: gustavoavena
Differential Revision: D56830459
fbshipit-source-id: 62145b8d3972d8c1b3263fb31e3559390d1f58ba
Summary:
I noticed when re-importing `whatsapp/voip` (which has ~2k tags) that even though most tags were already imported,
it took multiple seconds to iterate over each one with `generate-bookmark`.
Looking at the code, we used to upload the git tag objects every time, even if the tag
already existed.
That's at least two blobstore puts, which explains the delay.
After this change, the speed is closer to ~5-10 tags per second, as expected.
We could still optimize further by parallelizing the bookmark generation, but this diff
only fixes an obvious pessimization.
Reviewed By: YousefSalama
Differential Revision: D54684266
fbshipit-source-id: 5cddcbc7164a84f81fba184e0d83f6226d41a035
Summary:
The diff that was reverted broke one the tests. I believe CI didn't catch it because it was disabled.
Anyway this diff undoes that backout but of course fixes the test.
Reviewed By: kmancini
Differential Revision: D56884252
fbshipit-source-id: 9a54c2305a3d85793b2c673865528541d8cac43e
Summary: Undo backout (this diff wasn't the one that introduced the regression)
Reviewed By: kmancini
Differential Revision: D56883957
fbshipit-source-id: 034eead51816bf5ca6ee97fecb15fdd40e8c1fb8
