Summary:
pass client info into thrift services (land service, derived data
service)
we need to extract the client info from the metadata and pass it in a
persistent header in thrift
Reviewed By: RajivTS
Differential Revision: D49640441
fbshipit-source-id: 512fcbde0ebe249770eac78f0766e0043f447a4a
Summary:
X-link: https://github.com/facebookincubator/velox/pull/6685
libmnl-static has not been used for fboss in centos8. It is not
available in centos9. fboss builds fine without limnl dependency
Reviewed By: srikrishnagopu
Differential Revision:
D49380161
Privacy Context Container: L1125642
fbshipit-source-id: e1382a9d04c0fc962b4083c4aa7edba6bc1e02ae
Summary: We ship Python 3.9+ everywhere, so now we can use `ZoneInfo` instead of `pytz`, [which is what the `pytz` developers recommend](https://github.com/stub42/pytz/blob/master/src/README.rst#issues--limitations).
Reviewed By: muirdm
Differential Revision: D49624253
fbshipit-source-id: 70f97ed350bde146197bb930c2280ed7c9b875d9
Summary: Recently we made changes to which Python version to prefer. In some cases, the `python3` binary that comes first in the path might be older than 3.8, which is the minimal Python version we support. This changes it so that 3.10 and 3.8 are preferred
Reviewed By: muirdm
Differential Revision: D49623917
fbshipit-source-id: ca1c17cf4694a90fc1f6e1641ab231cee6b7cae5
Summary:
I'm adding more test cases to cover the partial_commit_graph logic in the next diffs and the file was starting to get too big...
I prefer having smaller files because I can open related code side-by-side and more granular crates to improve buck build (see [the wiki](https://www.internalfb.com/intern/wiki/Rust-at-meta/Buck/rust-in-fbcode/#optimizing-build-time)).
Differential Revision: D49634613
fbshipit-source-id: bfd78f51b5ba970379ad6cb9404856f36705c4c7
Summary:
When I set up the unit tests, I didn't know that I could also use drawdag there. I thought it was available only for integration tests.
Drawdag makes everything more succint and easier to understand/maintain.
Differential Revision: D49536292
fbshipit-source-id: a3786e1d74f1f802ab1ff29bca15b2cd6c6fd610
Summary:
To display progress when processing very large/old directories.
I'm only displaying the progress bar when the log level is set to INFO or lower, so it doesn't show up on the integration tests.
Differential Revision: D49452508
fbshipit-source-id: 17988c9140df6f1219fd63dd8ae305a6b0aa5ba0
Summary:
## Bug
See D49366697 and D49369883 for context on the bug.
## Solution
The comments on the code should explain the solution (please LMK if that's not the case).
The **TL;DR** is that any files that are copied in a changeset `A` should only be copied from a changeset that are is being exported to the git repo, and thus have been rewritten before `A` and will be present in the `remapped_parents` map.
## What's next
Stop relying on the temporary hack of manually passing the name of the old directory as an export path by automatically getting the name of the old directories and all the changesets that affected it.
Differential Revision: D49416858
fbshipit-source-id: d084427a2db0c835ddb6ba39c4148bd2a4df82f7
Summary:
the `Config` usages were deleted from D49593287, let's remove the
unused dependency as well
Reviewed By: muirdm
Differential Revision: D49605511
fbshipit-source-id: 77921c814e9d94036d39f9e6773bfb6165bc5875
Summary:
## Context
D49366697 adds an integration test to cover a bug that was surfaced when running the tool in a real directory.
The **TL;DR** of the bug is that gitexport crashes when we pass `foo` as the export directory and at some point in history `foo` was created by renaming an entire directory named `bar`.
## Temporary solution
Manually pass `bar` as an export directory.
However, this doesn't work out of the box because of the way that the `paths_with_history` method works.
The documentation of `paths_with_history` says that it will not basically ignore any paths that don't exist in the changeset from which the method was called.
This means that passing `bar` as an export directory has no effect, since it doesn't exist in the starting changeset anymore.
By calling `path_with_history` for each path, we'll get the changesets affecting the old directories, even if they don't exist in the starting changeset anymore.
This unblocks the fix in D49416858.
Differential Revision: D49369883
fbshipit-source-id: dc3a44752cdcf76f03efdcd5e0e76a1435972026
Summary: Recently we changed the way we pick the preferred Python version. This affected parts of our OSS build, which this diff fixes.
Reviewed By: muirdm
Differential Revision: D49612380
fbshipit-source-id: ea39893f992025c2ad97c24eff03bc5dfe7b2581
Summary:
This skips the ISL build and the "other" Rust binaries (scm_daemon and mkscratch).
I tweaked things so "make oss" shares the same body as "make local".
Reviewed By: quark-zju
Differential Revision: D49605727
fbshipit-source-id: a29213a059e11876454b302324c1c5d7752feabe
Summary:
X-link: https://github.com/facebookincubator/velox/pull/6734
This is phase 1 of "sapling" rename. Later we will move out of the "eden" directory.
These were the high level steps I took:
1. "hg mv edenscm sapling" + "hg mv edenscmnative saplingnative"
2. s/edenscm/sapling/g
3. s/EdenSCM/Sapling/g
4. Update autocargo config
5. Update mercurialshim.py to support "edenscm" as legacy module (this is the critical part that keeps backwards compat w/ extensions and hooks that import from "edenscm")
Reviewed By: sggutier
Differential Revision: D49326692
fbshipit-source-id: 843db9ece03c7b80c666ff805bd855211614ac62
Summary:
When error messages are bouncing between processes it is hard to keep things categorized. For example, if "hg checkout" calls into EdenFS which calls back into the hg import helper, the "is-network-error" knowledge is long lost.
Add fallback detection that turns UncategorizedNativeErrors into TlsErrors if the error message looks cert related.
Reviewed By: zzl0
Differential Revision: D49607587
fbshipit-source-id: 51f72feda2c970feb5b7178ebcddf4c3f22a9eff
Summary: We previously had counters to provide visibility into the in-memory blob metadata cache's hit rate, but we lacked corresponding counters for the in-memory blob and tree caches. This adds BlobCache hit counters.
Reviewed By: genevievehelsel
Differential Revision: D49564626
fbshipit-source-id: 2fdd1e51fb8944607a032e0aef8610b9bf8dabcf
Summary: Before this change there was a bug where trying to read symlinks with POSIX-style paths (e.g., `/foo/bar`) would fail. This applied both to reading symlinks directory or trying to list the contents of their parent directories.
Reviewed By: genevievehelsel
Differential Revision: D49601495
fbshipit-source-id: 403d06c580d676d61fb6bd8233e5ac45b072c2df
Summary:
We want to be able to understand how much time is being spent fetching data from EdenAPI. Let's add a time field to FetchMetrics so we can record time spent doing each fetch.
Disclaimer: EdenAPI stats only tracks the time spent fetching each BATCH, not each request. Therefore we can't get a per-fetch request metric, only a per-batch-fetch metric.
In addition, the ODS timer will record the total amount of time spent doing fetches for ALL fetch occurrences. To get an "average" fetch time, you'd have to divide the time by the total number of keys (or requests, if you want a per-batch number).
Reviewed By: quark-zju
Differential Revision: D49521776
fbshipit-source-id: cb3fa1a01c077284f8b3959d8cfb7e72708d4b27
Summary: We were previously not counting EdenAPI hits/misses in our FileStoreFetchMetric data. Let's include that data so that we have an idea of how often we're making requests to EdenAPI and how often we're "missing" or getting errors. EdenAPI can't really "miss", so any non-hit results are errors.
Differential Revision: D49521775
fbshipit-source-id: 6f50f62db8caf27060320b4d1aedde47fd59b9d7
Summary:
intializae ClientInfo with default ClientRequestInfo, let's use this
to do integration tests. The following diff will make it support config
Reviewed By: genevievehelsel
Differential Revision: D49558716
fbshipit-source-id: d90dbe59c1716b624a3f6c546fbc7fc67db1be16
Summary: expose it to Python world, so we can use the same global static variable.
Reviewed By: liubov-dmitrieva
Differential Revision: D49556642
fbshipit-source-id: d4e31d2f54def60810a72614153a85edc8e465c4
Summary: add a static global client request info object, we will use it for storing request info
Reviewed By: liubov-dmitrieva
Differential Revision: D49548930
fbshipit-source-id: 330eed86fa7179dd590e52358d3dac5c80c683f5
Summary: we will add more code for client request info, so let's move it to a separate file
Reviewed By: liubov-dmitrieva
Differential Revision: D49521918
fbshipit-source-id: c765dedfab4136e952e8c8127d1dd35bc79bb009
Summary: It's used only for one feature we have never used. The u64token structure in ClientInfo. Let's get rid of it because it will make passing and creating the ClientInfo through FFI boundaries easier in following diff.
Reviewed By: liubov-dmitrieva
Differential Revision: D49593287
fbshipit-source-id: 43438ac43040f199be19395827348a39d420c1f7
Summary:
We previously had counters to provide visibility into the in-memory blob
metadata cache's hit rate, but we lacked corresponding counters for the
in-memory blob and tree caches. This adds TreeCache hit counters.
Reviewed By: chadaustin
Differential Revision: D49396499
fbshipit-source-id: 5622874b8df9f8aece61df24989a1a12c9ff4331
Summary:
Adding an integration test case to repro the issue currently blocking gitexport.
The problem is that, even though we properly set the new parents in the bonsai changeset, there are still references to the old parents in the `copy_from` field of the `TrackedFileChange`.
Example: we export directory `foo` and have the following 3 changesets `A <- B <- C`, where
- A changes the directory `bar`
- B changes a random directory
- C renames `bar` to `foo`
This would crash because `C` will have file changes referencing `B`.
This will be fixed in the next few diffs by properly updating those references the same way we update the parents.
**Note:** I added two calls to `gitexport` on the test: one passing the old path name manually and another without doing that. That's because the former will work with some small tweaks that will unblock the use of the tool while the proper fix is under development.
Differential Revision: D49366697
fbshipit-source-id: c13b76abcee5f780c4a0f46e65e047ef4f8745d8
Summary:
Adding multiple logs and changing the severity of the existing ones to make sure some progress is shown and it's still possible to debug small repros in detail.
- All `info` logs are supposed to show progress on a high level to the end user.
- `debug` logs show progress in very granular detail, but can still be used when running the tool for a large directory in production.
- I'm using `trace` for the logs that scale based on the size/age of the directory (e.g. printing commit messages, sorted changeset vector).
Differential Revision: D49370152
fbshipit-source-id: a56f0ece417f085153fc4c49046f72951eef0607
Summary:
## What
Expose an argument for a timestamp that sets the lower bound of the commits. It's just exposing the one from `ChangesetPathHistoryOptions` so that we're able to export history only up to a specific date.
Differential Revision: D49230397
fbshipit-source-id: dad7b78d6a9b88072c13cd9c409fd5e0b9d56c92
Summary:
## What
Expose an argument to specify which changeset to start the history search from. We currently use the master bookmark, which will remain the default behaviour if one isn't provided.
I'm also adding some `info` logs that will be helpful to track the progress of the tool.
## RFC
I had to add the `BonsaiSvnrevMapping` trait to the `Repo` definition of MononokeAPI in order to use the `parse_commit` function in the CLI (which I extracted in D48896898).
Can that be a problem? None of the tests failed, so there shouldn't be any impact on business logic, but just wondering if there are other consequences that I'm not aware of... Asking because I imagine that the `Repo` in Mononoke API is a dependency of many other things.
Differential Revision: D48954151
fbshipit-source-id: c3a815bccd3d8372f2ff0caacd7b06e9f8c9456f
Summary: Build and install the isl-dist.tar.xz so `hg isl` can use it on *nix platforms.
Reviewed By: sggutier
Differential Revision: D49560037
fbshipit-source-id: 3664707e371cd58b97f4c702c6cc36c1642e5aa9
Summary:
Update the pick-python logic so it:
- Prefer `PYTHON_SYS_EXECUTABLE` if set.
- Support missing `CFLAGS` in sysconfig. It's not set on Windows.
Reviewed By: sggutier
Differential Revision: D49560365
fbshipit-source-id: 96327157593a65bbc2c2b0475ebceff11aff4bce
Summary:
Let `make local`, `make oss` that use `setup.py` generate the `isl-dist.tar.xz`
in the `setup.py` directory.
Reviewed By: sggutier
Differential Revision: D49558559
fbshipit-source-id: 813f4843adfaef95d3b76b141c86f8991e647a50
Summary:
Change ISL logic from using dotslash and (OSS) Javascript source tree to the
`isl-dist.tar.xz`. This is one step towards unifying OSS, internal non-buck,
and buck builds.
The locations of `node` and `isl-dist.tar.xz` are configurable.
Reviewed By: sggutier
Differential Revision: D49558560
fbshipit-source-id: 6f19f66ae31081ca1cfc79290c2cfe219b47ebe4
Summary: The data local or cache directories are handy in the Python world too.
Reviewed By: sggutier
Differential Revision: D49558557
fbshipit-source-id: 5f57a4e89ba88a68cc0b91c332fed57b74788956
Summary:
This script takes the ISL source, uses `yarn` to build, then write the built
result to a tarball. It maintains a "source_hash" and skips building if the
destination "source_hash" does not change.
The yarn build commands and directories to include are from `isl/release.js`.
Reviewed By: evangrayk
Differential Revision: D49558562
fbshipit-source-id: 060fec91a1f65cda51efa525ec291cd8483cf42b
Summary:
Add logging for `hg edendebugimporter`
HgBackingStore will:
- Attempt to get the data from local cache
- Fetch it remotely from Mononoke
- Or, as a last resort, fall back to using `hg edendebugimporter`. This is a path we would like to deprecate and logging accesses will help us understand who is falling back to this.
Reviewed By: jdelliot
Differential Revision: D49424377
fbshipit-source-id: 959c6a42bb5a5c3df6b9bc206527d0ed28253bbe
Summary: During multi-step operations like rebase, we move the working copy to different parents. Previously the Rust workingcopy's "status" calls to EdenFS were not using an up-to-date parent id in the "status" args, resulting in the "requested parent commit is out-of-date" error. Fix by pulling the parent id out of the treestate for every "status" call.
Reviewed By: quark-zju
Differential Revision: D49509139
fbshipit-source-id: 7dff48e826c073897c5fd1f683ac76f2f0068753
