Summary:
The PathFuncs.h header is included pretty much everywhere in EdenFS, and thus
its compile speed affects the entirety of EdenFS. Unfortunately, the
boost::filesystem dependency is pretty large and is thus slowing down the
compilation.
Since the few methods that rely on Boost are only used in one place, let's
instead inline the use of Boost in these places.
Reviewed By: chadaustin, fanzeyi
Differential Revision: D35058316
fbshipit-source-id: b80a39f1575acc0fa4b9e64eb5375b95b4b421da
Summary: Untested code breaks silently. Tested code also breaks, but we get to know exactly when it happens.
Reviewed By: yancouto
Differential Revision: D34899023
fbshipit-source-id: b50fd4ff1c75853bdd3dae443160d8f630fdb5be
Summary: For mutable renames tests, I'll need copy-from commits. Make drawdag generate them, because it's awesome to not have to run a sequence of Mercurial commands
Reviewed By: Croohand
Differential Revision: D34898156
fbshipit-source-id: f5e73fca63ae391d934b73da6cd3680179b4cd34
Summary: When you add a new mutable renames entry to a commit, we should copy over immutable renames automatically, then override with the newly added one.
Reviewed By: Croohand
Differential Revision: D34893586
fbshipit-source-id: cde687a1e84979f2a57ced4d9e57f0de918d30cc
Summary:
For commits using mutable renames, we don't inspect immutable renames.
Make it possible to copy immutable renames to mutable renames, overwriting any overlapping mutable renames
Reviewed By: Croohand
Differential Revision: D34893588
fbshipit-source-id: 8fde522852b9390be7593ec8f29c36357bcf34de
Summary: Make it easy to add mutable renames by admin intervention. This does recursion into subdirectories automatically, so that one command can be used to add mutable renames markers for an entire directory tree.
Reviewed By: Croohand
Differential Revision: D34893587
fbshipit-source-id: e126bbcc507ec0bdb366f6b9caa90fdff64c602d
Summary:
EdenFS doesn't want to use retries while it has the hg import
fallback. Let's allow disabling them. Once there is a config that controls the hg
import fallback, we can enable retries when the fallback is disabled.
Reviewed By: xavierd
Differential Revision: D35020586
fbshipit-source-id: d51d7fb70d2627e55c1170f130c548d7b14a8e6e
Summary:
The diff includes the following key changes:
- Bounded the range of keys for enumeration in `PrefixBlobstore` by prepending the `prefix` for both empty and non-empty keys.
- In case of empty `end` keys, added `prefix + ascii_end_character` for limiting the search results to the prefixed namespace.
- Fixed enumeration in `ManifoldBlobstore` by expanding the struct definition to include a user-specified `directory`. In case of `None`, the `flat` directory (i.e. no directory) is used.
- The input keys for `get`, `put`, `is_present`, `copy` and `unlink` are now first qualified with the directory path before being used for the relevant operations in Manifold and ThriftManifold blobstores.
- Updated `ManifoldBlobstore` and `ThriftManifoldBlobstore` creation sites to explicitly pass the directory or `None`.
Reviewed By: farnz
Differential Revision: D34996443
fbshipit-source-id: d7b7a7d034b3cce15f64472a08f415494fda2746
Summary:
I got a task about a flaky LFS server test being disabled. The failure
was easy to reproduce when running 10 concurrent tests in opt mode.
The test was verifying that the LFS server will fetch content from its own
internal store if the upstream server (in this case netcat that sleeps for a
second) is slow to respond.
The flakiness was because sometimes the upstream server was quicker to respond
than checking our own internal store.
After some debugging, I found that this was because the server was failing to
lookup the name of the upstream. This meant that the test was comparing whether
it was faster to fail a DNS lookup compared to checking internal storage for
content.
```
+ V0322 12:39:45.061522 2262733 [tk] eden/mononoke/lfs_server/src/batch.rs:151] upstream_objects, request_id: 5afbf
+ V0322 12:39:45.061622 2262733 [tk] eden/mononoke/lfs_server/src/batch.rs:161] upstream_objects pre res, request_id: 5afbf
+ V0322 12:39:45.061681 2262733 [tk] eden/mononoke/lfs_server/src/lfs_server_context.rs:378] upstream: http://:48155/objects/batch, request_id: 5afbf
+ V0322 12:39:45.061839 2262733 [tk] eden/mononoke/lfs_server/src/batch.rs:301] internal_objects, request_id: 5afbf
+ V0322 12:39:45.062321 2262733 [tk] eden/mononoke/lfs_server/src/batch.rs:571] batch: internal ready, request_id: 5afbf
+ V0322 12:39:45.062728 2262733 [tk] eden/mononoke/lfs_server/src/batch.rs:163] upstream_objects: res: Err(UpstreamBatchNoResponse(An error occurred forwarding the request to upstream
+
+ Caused by:
+ 0: error trying to connect: dns error: failed to lookup address information: Name or service not known
+ 1: dns error: failed to lookup address information: Name or service not known
+ 2: failed to lookup address information: Name or service not known)), request_id: 5afbf
+ V0322 12:39:45.062896 2262733 [tk] eden/mononoke/lfs_server/src/batch.rs:589] batch: upstream ready, request_id: 5afbf
```
This change was introduced in D34923381 (acb8e3725a), which updated `${LOCALIP}` to
`${mononoke_host}`. However, `LOCALIP` is a variable name whilst
`mononoke_host` is a function name. Update the syntax to reflect this.
Reviewed By: mzr
Differential Revision: D35046184
fbshipit-source-id: 5ec9c29e2e9e8658ddabe29c755debdfb3a49938
Summary:
Our use of RCU is fundamentally incompatible with N4895 since it allows a read
section to be entered in one thread and closed in another one. We use this
feature in Prjfs to prevent a mount to be unmounted while requests are ongoing
on it.
Let's instead make use of a folly::AtomicReadMostlyMainPtr which is equivalent
to a folly::Synchronized<std::shared_ptr<>> without requiring as many atomics
in the read path.
Reviewed By: Decave
Differential Revision: D34982722
fbshipit-source-id: 299f25fc8c8aa72fb89cc9fc880860780ea0517d
Summary: Migrate blobstore_healer tool (which we use to continuously verify if all our underlying storages agree about contents of blobs and fix any discrepancies) to new framework.
Reviewed By: mitrandir77
Differential Revision: D34859855
fbshipit-source-id: 5814fd275f22113689afe25b55c28a9d5b062551
Summary:
On Windows, some users have experienced hangs during checkout, and their
backtraces points to a directory invalidation being run in the server state
thread pool. Most of these have been moved to a background thread in D33773892 (67e6d48b0d)
but one was missed: when removing an empty directory, a directory invalidation
is done in a blocking manner.
To fix this, this diff moves the location of where this removal is happening,
from the post checkout step, to the parent inode. This avoids having to make
the removal method async.
Reviewed By: fanzeyi
Differential Revision: D34930808
fbshipit-source-id: 76762dab5a8143ef37ca49cb6386e6df73e47d9f
Summary:
In order to add solana-client = "1.9.5" to third-party crates in fbcode, we need to update the versions of following crates.
- sha2
- zstd related including zstd, zstd-sys, zstd-safe
Next step is to add solana-client.
Reviewed By: wqfish
Differential Revision: D34648217
fbshipit-source-id: 719e499d9fbb4fd35624f3b9c46ffc6dfc94576d
Summary:
Sometimes we need to make a loud announcement to all users of a given repo.
(For example to tell them that their repo will go away soon). This allows us to
do it without further code changes.
Reviewed By: farnz
Differential Revision: D34934883
fbshipit-source-id: 771ee0c32d3435650a872efb7191350fdb40d12d
Summary:
Key changes in this diff include:
- Introduced the `ephemeral_store` new admin command for working with data stored within bubbles (snapshots for now).
- Implemented the outline for `list`, `cleanup` and `info` subcommands within `ephemeral_store`.
- Added deletion checks based on `BubbleDeletionMode` configuration flag within `ephemeral_store`.
Reviewed By: yancouto
Differential Revision: D34936192
fbshipit-source-id: 4f0b1f759112585ad165332e141687be0c39cedd
Summary: Add the requested URL to the HttpError details. This makes it clear what was being requested when we got an unexpected HTTP response.
Reviewed By: DurhamG
Differential Revision: D34907302
fbshipit-source-id: 18a7da77038b6c8acd1933465a31f5744ed4d8fc
Summary: Enums in rust are one of the ways the Rust compiler checks for edge cases, so I'd love to use them in my code.
Reviewed By: dalyadickstein
Differential Revision: D34932071
fbshipit-source-id: 4e5fd23d34e462f9e5a533c917704ace1afe99e1
Summary:
A EdenFS messenger dogfooder on macOS is hitting our check here that file
handles be a certain size.
This likely indicates that we have a big somewhere in parsing that is causing
us to parse something that is not a file handle as a file handle. Its also
possible this request could be malformed.
Eventually we should not trust clients enough to allow them to crash eden with
a malformed request, but the first step to fixing this bug is understanding
wether the request that crashes us is malformed or not.
So first I am adding some extra logging info to this xcheck so that we can
determine the cause. I will follow up with a solution to prevent clients
from crashing EdenFS.
Reviewed By: xavierd
Differential Revision: D34969740
fbshipit-source-id: 783ca1147f6c9d9c86996836d14263a0a79d6b7d
Summary:
Tuple structs as part of public interfaces are hard to modify, and result in
`.0` appearing all over the code. Make the field private and add well-defined
methods with appropriate visibility.
Reviewed By: farnz
Differential Revision: D34971449
fbshipit-source-id: 3078469527e4021dceef5dcb6a5f5f2f980d836a
Summary:
`mononoke_admin create-bonsai` is a dangerous command as it does not check the
bonsai is valid. It is used in tests to test Mononoke correctly handles
invalid bonsai changesets.
Move it to the test tool so it is clear that it is only for use in tests.
Reviewed By: farnz
Differential Revision: D34892793
fbshipit-source-id: dd11b9e33920d06883258c1051eadc98e9ea3d81
Summary: Port the split-commit command to the new admin tool (as `newadmin commit split`).
Reviewed By: yancouto
Differential Revision: D34864663
fbshipit-source-id: 303b1ef568a903f38d8eff33924c5d8a750ee48a
