Summary: This diff makes `InodeBase::stat()` to be able to accept `ObjectFetchContext` as a parameter.
Reviewed By: chadaustin
Differential Revision: D21780883
fbshipit-source-id: 9b1db2e2268cb98663bcf902ea61897da593ea05
Summary: This diff starts to use `RequestData` as `ObjectFetchContext` in our `read()` methods. This ensures EdenFS could track backing store fetches happened in FUSE requests.
Reviewed By: chadaustin
Differential Revision: D21792503
fbshipit-source-id: 9509a1bc8f28100a0dfe196e312c4785c7842345
Summary:
Previously we check if a request is a fuse request when we fetch anything from backing store, so we can collect number of fetches happened for each process in eden top.
This is creating a dependency from store to fuse, which is a little awkward. Instead, we could make `RequestData` a `ObjectFetchContext` and record the fetches when that happens.
Similarly in the future we should also have something equivalent in our Thrift layer.
Reviewed By: kmancini
Differential Revision: D21775919
fbshipit-source-id: 95056830ddbe7c999051c43e0d8eca9a67350904
Summary:
All of these are instances of `CommitSyncer`, named `commit_sync_config`. It
makes a reader think they may be instances of `CommitSyncConfig`.
Reviewed By: farnz
Differential Revision: D22113917
fbshipit-source-id: bb682b24d7cadf87ca7721f341dbc8231636c99a
Summary: Megarepo is simplified if we can avoid copying hooks everywhere - run megarepo hooks as well as small repo hooks during pushredirection.
Reviewed By: StanislavGlebik
Differential Revision: D20652331
fbshipit-source-id: f42216797b9061db10b50c1440253de1f56d6b85
Summary:
Tooling can't handle named_deps yet, but it can warn about them
P133451794
Reviewed By: StanislavGlebik
Differential Revision: D22083499
fbshipit-source-id: 46de533c19b13b2469e912165c1577ddb63d15cd
Summary:
Remove unused dependencies for Rust targets.
This failed to remove the dependencies in eden/scm/edenscmnative/bindings
because of the extra macro layer.
Manual edits (named_deps) and misc output in P133451794
Reviewed By: dtolnay
Differential Revision: D22083498
fbshipit-source-id: 170bbaf3c6d767e52e86152d0f34bf6daa198283
Summary:
Two changes here:
1. The `[patch.crates-io]` section of `third-party/rust/Cargo.toml` is being now copied over to workspaces generated by autocargo for OSS and in the runtime generated Cargo.toml file for cargo-fbcode builds. Without that some projects could be buildable in Buck internally, but not externally on GitHub due to missing patches.
2. If a `[workspace]` Cargo.toml file is being generated and there is already a generated Cargo.toml file in the same directory then instead of overriding that file the `[workspace]` (and `[patch]`) sections are appended to that Cargo.toml file.
Reviewed By: farnz
Differential Revision: D22023144
fbshipit-source-id: dec54491c36c2ee0ab29eefb722b3eceaef6ffe1
Summary:
The goal of the stack is to support hot reloading of `CommitSyncConfig`s everywhere: in `push_redirector`, `backsyncer`, `x-repo sync job` and so forth.
This diff in particular is a refactoring of how we instantiate the `PushRedirector` struct for the `unbundle` flow. Previously the struct would be instantiated when `RepoHandler` struct was built and would later be reused by `RepoClient`. Now we want to instantiate `PushRedirector` before we start processing the `unbundle` request, so that we can request the newest `CommitSyncConfig`. Note that this diff does not introduce the hot reload itself, it just lays the groundwork: instantiation of `PushRedirector` at request start.
To achieve this goal, `RepoClient` now contains a somewhat modified `PushRedirectorArgs` struct, whose goal is to own the unchanging stuff, needed to create a full `PushRedirector`.
Here are a few explicit non-goals for this hot reloading:
- the overall decision whether the repo is part of any `CommitSyncConfig` or not is still made at `RepoHandler` creation time. What this means is that if `CommitSyncConfig` is changed to have another small repo and Mononoke servers happens to know about that repo, it would not automatically pick up the fact that the repo should be a part of `CommitSyncConfig`
- same for removal (stopping push redirector is already possible via a different hot-reloaded config)
- changing anything about a large/small relationship is likely to be very complicated under the best circumstances of everything being down, let alone during a hot reload. This means that target repo cannot be changed via this mechanizm.
Essentially, the goal is only to be able to do a live change of how paths change between repos.
Reviewed By: StanislavGlebik
Differential Revision: D21904799
fbshipit-source-id: e40e6a9c39f4f03a436bd974f3cba26c690c5f27
Summary: Added basic main and target files for the repo import tool
Reviewed By: StanislavGlebik
Differential Revision: D22067642
fbshipit-source-id: 5ed97e3f2446c86314918f57103c2ecb911e30b0
Summary:
Add optional compress on put controlled by a command line option.
Other than costing some CPU time, this may be a good option when populating repos from existing uncompressed stores to new stores.
Reviewed By: farnz
Differential Revision: D22037756
fbshipit-source-id: e75190ddf9cfd4ed3ea9a18a0ec6d9342a90707b
Summary: Add zstd decoding support to packblob so that if store contains individually zstd compressed blobs we can load them on get()
Reviewed By: farnz
Differential Revision: D22037755
fbshipit-source-id: 41d85be6fcccf14fb198f6ea33a7ca26c4527a46
Summary:
Add a regex for repo prefix detection, and use it in prefixblob for removal of repo prefix from blob-embedded keys.
This is important to keep blobs copyable between repos, and to allow dedupe between same blob in two repos.
I looked at alternate approaches of passing down the prefix from PrefixBlob::new(), but that was fragile and didn't cover the use cases from things like blobstore_healer where no prefixblob is configured at all but packblob will be in the blobstore stack. Using a pattern is the only real option in such "all repo" cases.
The aim of binding the pattern and its tests closely to the prefix generation is to make it hard for someone to get them out of sync, and provide a clear local test failure if they do.
Reviewed By: farnz
Differential Revision: D22034638
fbshipit-source-id: 95a1c2e1ef81432cba606c22afa16b026f59fd5f
Summary:
Add pack and unpack logic to packblob.
Loading a packed form is possible from a regular get(), as the store may contain packed data.
Storing a packed form is only possible from a new put_packed() method, intended for use from the packer (and tests).
NB As noted in the TODO in get, this does not yet handle prefix removal on get, will address that in a separate diff.
Reviewed By: StanislavGlebik
Differential Revision: D21980498
fbshipit-source-id: f534b0e754aa29c42bf00bb7e764f93f1446c14c
Summary: Add blobstore link trait so we can use hardlink style links in fileblob and memblob for testing and later sqlblob et al for prod.
Reviewed By: StanislavGlebik
Differential Revision: D21935647
fbshipit-source-id: f76eaca26b6b226c77d6e39e9c64e02b4145b614
Summary: Fix the capitalization in the error message to match the other variants.
Reviewed By: singhsrb
Differential Revision: D22078125
fbshipit-source-id: 4a1de0111fa4dceea2d401c47ef145669a3cc6f5
Summary: asyncify get_file_history_using_prefetched. Initially I was planning to change this function in the next diff, but then changed my mind. However asyncifying this function won't hurt to land.
Reviewed By: krallin
Differential Revision: D21881121
fbshipit-source-id: 0b2e92ae015b72e1f4578153c56708faabc4bf13
Summary: this is the last diff to complete the `reachabilityindex` module migration
Reviewed By: farnz
Differential Revision: D22049853
fbshipit-source-id: d8510f2d1b1c12a1ba903c2afda37096aadd8347
Summary:
When inserting multiple rows using `INSERT IGNORE`, it's possible to hit
deadlocks if two competing transactions insert the same rows in a different
order.
This can be triggered in the mutation store if two clients are pushing
overlapping sets of commits, and the arbitrary changeset order from the
`HashSet` gives the commits in a different order. For example the first
transaction might be inserting "A, B, C", be holding the lock on row A, and be
waiting for the lock on row C, meanwhile another transaction is inserting "C,
D, A", holding the lock on row C, and waiting for the lock on row A.
To avoid these deadlocks, always insert the mutation changesets in a stable
order (sorted by changeset hash or successor hash).
Reviewed By: krallin
Differential Revision: D22067338
fbshipit-source-id: aa54c8a9c0cac4e4ae35855b44e759f8b6cb4f59
Summary:
Add logging of the repo root and repo shared root to the sampling extension.
Reduce the number of different strings that are logged by replacing any path
component that equals the user's username (e.g. their home directory) with
`$USER`.
Reviewed By: farnz
Differential Revision: D22066053
fbshipit-source-id: 39221e5182b3d4df51afa386497003e637a50b13
Summary:
async the blobstore test functions, in preparation for adding test coverage for BlobstoreWithLink::link
The boxable test was duplicating the simple roundtrip test which seemed unnecessary, now it just checks that we can Box the blobstore type.
Reviewed By: krallin
Differential Revision: D21974471
fbshipit-source-id: a3cd71a7503a3a670b3c6223812a2e870a20a16e
Summary: update memblob imports to futures_old in preparation for using new futures to implement BlobstoreWithLink
Reviewed By: krallin
Differential Revision: D21973642
fbshipit-source-id: dea4a686db41a0d618395c4dc411603f922a9e22
Summary: Update imports in preparation for using new futures for the BlobstoreWithLink mixin trait
Reviewed By: StanislavGlebik
Differential Revision: D21973587
fbshipit-source-id: 094aa82abc21848d43f50d6d169087e19339ba04
Summary: Switch blobstore imports to use futures_old naming in preparation for using new futures in BlobstoreWithLink (and later BlobstoreWithEnumeration) traits.
Reviewed By: krallin
Differential Revision: D21972842
fbshipit-source-id: 6022c98e7b7254f7c7beb46dc1c7f83609810853
Summary: Add basic packblob store which wraps blobs in the thrift StorageEnvelope type on put() and unwraps them on get()
Reviewed By: farnz
Differential Revision: D21924405
fbshipit-source-id: 81eccf5da6b5b50bdb9aae13622301a20dca0eac
Summary:
Thift definitions for a new blobstore layer known as packblob, which is an enabling part of compression at the mononoke blobstore layer. Its storage formats will support raw data, individual blob compression and blob delta compression.
The included README.md has more detail on the expected use of these types.
Reviewed By: farnz
Differential Revision: D21761723
fbshipit-source-id: a277800f4b4aed5972c4a5bb277e984ff12660f8
Summary:
I'm going to change it in the next diffs, so let's migrate them to new futures
first.
Reviewed By: krallin
Differential Revision: D22065216
fbshipit-source-id: b06fcac518d684f40b1d19fcef9118ca51236873
Summary: Without this, the most lagged shard will slow down all shards; with this, a single lagging shard doesn't slow down other shards.
Reviewed By: ahornby
Differential Revision: D22018264
fbshipit-source-id: c23f25ea1d0b0b6891d2a6f76dfed467101d2f4d
Summary: I'll want the delay operation to respect shard_id - move it into the stores. While here, do some refactoring that will reduce the size of the next diff.
Reviewed By: ahornby
Differential Revision: D22018317
fbshipit-source-id: d2333dfc6749b2a13bc8a67bfa953ed25cedf847
Summary:
test_no_units_are_active was failing on CentOS 8. Presumably systemd
has some new default services. I believe genevievehelsel is planning on
replacing all of this code, so, rather than understand the failure,
just ignore the additional service names.
Reviewed By: genevievehelsel
Differential Revision: D22055992
fbshipit-source-id: b6f313350d0e1b107fe6ea3d7ed7f5b4eb025ef3
Summary:
From a simple `hg trace log -r .`, it appears that importing the module takes
16ms, which dominates the time it takes to load the treemanifest extension.
Since the module isn't necessary, don't import it.
Reviewed By: farnz
Differential Revision: D22048075
fbshipit-source-id: ccaf2d53eb006faa36d97df452b0db10f4983c23
Summary: Prep for 1.44 but also general cleanups.
Reviewed By: dtolnay
Differential Revision: D22024428
fbshipit-source-id: 8e1d39a1e78289129b38554674d3dbf80681f4c3
Summary:
Python 3.8 provides the (poorly named)
unittest.IsolatedAsyncioTestCase base class that gives each test an
asyncio event loop and runs it. This allows us to write async
integration tests, required to use modern Thrift Python.
Reviewed By: genevievehelsel
Differential Revision: D21955584
fbshipit-source-id: 8366dc7c72b31e1667e72fe1e359587b74e2c614
