Summary:
This removes the Extend implementation for FileBytes, which was incorrect (it
discarded existing data!). I had introduced this as a backwards compatibility
shim when doing the Bytes 0.4 to Bytes 0.5 migration :/
We don't really need this shim, considering:
- The only place that really matters that uses this is the remotefilelog crate,
where we have a content id, and where we should use `filestore::fetch_concat`
instead.
- The other places are tests (or close to abandonware...), which can do their
own folding.
Longer term, I'd like to remove the whole `Content` stream in hg entries, so
those callsites can use the filestore methods, which a) have test coverage
(unlike ad-hoc folds, which don't always do), and b) are more efficient since
they know how large the destination buffer needs to be ahead of time, and don't
need to re-allocate.
To make sure this fixes the bug, I also introduced tests for the remotefilelog
crate. As expected, the chunked variant fails without this fix.
Reviewed By: mitrandir77
Differential Revision: D20248978
fbshipit-source-id: 1b554d3e595eb867b6b6cf4204d31f27dd90a111
Summary:
Not sampling errors will make it easier to use Fastreplay as an early alarm
system for errors.
Reviewed By: ahornby
Differential Revision: D20249202
fbshipit-source-id: 92da53d5703b58bcef49cfcdc251f008ae6f25bc
Summary:
The change is in theory not necessary. However it improves the reliability on
OS crashes a bit, and can potentially workaround some bugs in filesystems
(as we saw in production where the atomic-written files are empty and the
system didn't crash).
The idea is, the `symlink` syscall does the file creation and "content" writing
together, while there is no way to create a file and write specific content
in one syscall. Note that the C symlink call uses 0-terminated string, and
the Rust stdlib exports it as accepting `Path`. To be safe, we encode binary
or non-utf8 content using `hex`.
For downgrade safety, the write path does not use symlink by default unless
format.use-symlink-atomic-write is set to true. This makes downgrade possible:
the read path is rolled out first, then we can turn on and off the write path.
The indexedlog Rust unit tests and test-doctor.t are migrated to use the new
symlink code paths.
Reviewed By: DurhamG
Differential Revision: D20153864
fbshipit-source-id: c31bd4287a8d29575180fbcf7227d2b04c4c1252
Summary:
This makes it possible to implement atomic_write differently (ex. use a
symlink).
Reviewed By: DurhamG
Differential Revision: D20153865
fbshipit-source-id: 07fa78c2f2dac696668f477c75f65cf70950b73f
Summary:
The git mappings are normally populated during blobimport of the repo but we
need something for the repos we've already imported.
Reviewed By: markbt
Differential Revision: D20160768
fbshipit-source-id: 9e37c7d0f12682e73ca9990e56e4d827e9861a9f
Summary:
We don't use it, and this tries to write to Manifold from tests, which is
undesirable. Let's remove it;
Reviewed By: farnz
Differential Revision: D20219902
fbshipit-source-id: 2e983bee54cadad257648cc9633695be825a1ef3
Summary:
This introduces a new binary and library that (microwave: it makes warmup
faster..!) that can be used to accelerate cache warmup. The idea is the
microwave binary will run cache warmup and capture things that are loaded
during cache warmup, and commit those to a file.
We can then use that file when starting up a host to get a head start on cache
warmup by injecting all those entries into our local cache before actually
starting cache warmup.
Currently, this only supports filenodes, but that's already a pretty good
improvement. Changesets should be easy to add as well. Blobs might require a
bit more work.
Reviewed By: StanislavGlebik
Differential Revision: D20219905
fbshipit-source-id: 82bb13ca487f82ca53b4a68a90ac5893895a96e9
Summary:
The walker has been hitting the filenodes-enforced 5 second SQL timeout when
querying filenodes from MySQL.
It's not clear why that is, but looking at previous run history shows that we
occasionally have queries that take > 30 seconds to complete (none of those
show up in MySQL slow queries, though, and there's no particular load on the
hosts around that time, so it's not clear whether this is happening in MySQL or
our end).
Anyhow, those queries would have worked in the old implementation (after a long
time), but they fail in the new one, since it enforces a 5-second timeout.
We should investigate why this is happening (and Alex has landed diffs to add
more reporting in the walker to that end), but in the meantime, there's no
reason to break the walker
Reviewed By: farnz
Differential Revision: D20227842
fbshipit-source-id: 5ee5c8225b6474b66c1f48a10b4a2d671ebc79c6
Summary: When it fails, it's better to know which repo failed.
Reviewed By: farnz
Differential Revision: D20245375
fbshipit-source-id: 9794911308dbdd67b20673857ac8b7b54f06a217
Summary: Makes it easier to understand which repo is failing
Reviewed By: krallin
Differential Revision: D20244630
fbshipit-source-id: ca32f7831c5ed4e701103020e9878c459ba6d573
Summary:
If hghave fails to check a feature because the feature name is unknown, treat
it as a test failure instead of skipping the entire test. This is especially
useful since `#if feature-name` only affects part of the test and failing to
test the feature should not skip the entire test. It also allows us to capture
issues about mis-spelled feature names or stale feature tests.
This has bitten us twice in the past:
- D18819680 removed `pure` and accidently disabled tests including
`test-install.t`, `test-annotate.t` and `test-issue4074.t`. Those tests got
re-enabled as part of D20155399, while they pass Python 2 tests,
the Python 3 tests were failing.
- D18088850 removed svn related feature checks, which has caused some issues
that got fixed by D18713921 and D18713922.<Paste>
Reviewed By: xavierd
Differential Revision: D20231782
fbshipit-source-id: 6adf99bd79b2a295d4e84ce4da5f9425a100936a
Summary: The test should not assert Python version is "2.*".
Reviewed By: kulshrax
Differential Revision: D20231781
fbshipit-source-id: 2e10c37bb4b665bc4d5d4b27329c4c2cb23d54e3
Summary: Make these functions generic so that callers don't need to construct a trait object whenever they want to call them. Passing in a trait object should still work so existing callsites should not be affected.
Reviewed By: krallin
Differential Revision: D20225830
fbshipit-source-id: df0389b0f19aa44aaa89682198f43cb9f1d84b25
Summary: Add a method to `HgFileContext` to stream the history of the file. Will be used to support EdenAPI history requests.
Reviewed By: krallin
Differential Revision: D20211779
fbshipit-source-id: 49e8c235468d18b23976e64a9205cbcc86a7a1b4
Summary: Add an 'HgTreeContext' struct to the 'hg' module to allow querying for tree data in Mercurial-specific formats. This initial implementation's primary purpose is to enable getting the content of tree nodes in a format that can be written directly to Mercurial's storage.
Reviewed By: krallin
Differential Revision: D20159958
fbshipit-source-id: d229aee4d6c7d9ef45297c18de6e393d2a2dc83f
Summary: I was looking in the `edenfs_events` table and saw that sandcastle was logging to this table. Rice was able to identify that the reason was because the integration tests were logging. So if we're on running integration tests, we should return a `NullTelemetryLogger`. The daemon currently does not log on sandcastle AFAIK.
Reviewed By: simpkins
Differential Revision: D20203556
fbshipit-source-id: e09175347631478cb366d4fa2c6092d976504dd8
Summary: `buck run edenfsctl -- du --clean` would help reduce the space used by the storage engine.
Reviewed By: chadaustin
Differential Revision: D20200616
fbshipit-source-id: 6ffa588fc71660a6a80d81aef7d58dda08932374
Summary:
Add a `get_build_info()` method to the `EdenFSProcess` objects returned by
the `process_finder` module. This returns information about the process
version and build time.
Reviewed By: wez
Differential Revision: D20178487
fbshipit-source-id: b1eb41de9184ca59dc1e90d0a92ff1cbc89a6b77
Summary:
Store member variables in a local variable so that Pyre will allow unwrapping
it from an `Optional` type. Pyre refuses to allow member variables to be
extracted from `Optional` since other functions called indirectly could modify
them.
Reviewed By: fanzeyi
Differential Revision: D20212162
fbshipit-source-id: 95655b73b5e469688f48d402c0b587928cbb0a35
Summary:
This makes it clear that `log` is a math concept, not an append-only file like
`Log`.
Reviewed By: DurhamG
Differential Revision: D20149376
fbshipit-source-id: 67d2e9584b15f48759ca9b6dfce4279a5b1365a0
Summary: This makes it friendly to Python 2.
Reviewed By: sfilipco
Differential Revision: D20162233
fbshipit-source-id: 5beb7a0f52159afc454332ff6e37e13087177cc0
Summary:
When I run `hg doctor` in my www checkout it fails the assertion check of the
first line of visibleheads is "v1". Make it graceful so doctor can check and
fix other components.
Reviewed By: DurhamG
Differential Revision: D20147969
fbshipit-source-id: 6aee2cab962fcd0ef06a0611d288021e86621249
Summary: Updated find_eden to find the Eden clone on a Windows system. On Windows we don't use symlinks, which make the logic different than on POSIX implementation.
Reviewed By: simpkins
Differential Revision: D19953934
fbshipit-source-id: bfbc112c3ccc48735ec6590746d8275cc9850796
Summary:
In D20130406 I updated the CLI to call `getDaemonInfo()` to check on the
server status. However, some very old EdenFS instances do not have have this
method. These instances should all be gone shortly, but for now update the
code to handle the unknown method error and fall back to calling `getPid()`
and `getStatus()` separately.
I implemented this in our `EdenClient` wrapper class, similar to our existing
wrapper for `getPid()`.
Reviewed By: fanzeyi
Differential Revision: D20212518
fbshipit-source-id: 9d48bdd26822802a7e9776128c5567436d4bb445
Summary:
Update the import statements so that autodeps works on the `eden/py`
directories.
Reviewed By: fanzeyi
Differential Revision: D20212519
fbshipit-source-id: 37ccabf14dc0dbfe998664260ae9b83c9136ad63
Summary:
Update process_finder.py to look for a process's Eden state directory by
looking through its open FDs to find the EdenFS lock file, if it can't find
the state directory from the command line arguments.
At the moment we almost always invoke EdenFS with an explicit `--edenDir`
argument, but this code will allow this code to work even if we remove that in
the future.
Reviewed By: wez
Differential Revision: D20178484
fbshipit-source-id: 361b78f4a2566b8c09ce02fb21c46233d7e2546b
Summary:
Update the `FakeProcessFinder.add_edenfs()` function to also add a fake
privhelper process in addition to the main edenfs process. This allows the
tests to more accurately simulate the normal edenfs behavior.
Reviewed By: wez
Differential Revision: D20178482
fbshipit-source-id: edc70ade1b61929b37f13ece77757c7c35aa4eec
Summary:
Update the code in `process_finder.py` to return EdenFS processes owned by all
users. We now report the `uid` as a field in the returned process info, so
that callers can filter the results based on user ID if they want. This
allows callers more flexibility when finding processes.
This also updates the `FakeProcessFinder` test utility code to support
providing fake UIDs to allow testing this behavior.
Reviewed By: wez
Differential Revision: D20178490
fbshipit-source-id: 6b76e1109e4835b167c80688fd3ace50f7986a22
Summary:
Move the code to find rogue EdenFS processes out of the generic
`process_finder` module and into the `check_rogue_edenfs` module that is
specific to the `eden doctor` checks.
The `ProcessFinder` class now exposes a `get_edenfs_processes()` API instead
of `find_rogue_pids()`, which makes it more generically usable outside of just
the doctor code.
Reviewed By: wez
Differential Revision: D20178486
fbshipit-source-id: e289f1673a5d4a666e9d54e8f58f4f00bdde94b7
Summary: - added logging only around the import tree call to capture non-queue related wait time
Reviewed By: chadaustin, fanzeyi
Differential Revision: D20207472
fbshipit-source-id: d88bb34ce224a26ff2be100d7789ddeff608006d
Summary:
- added logging only around the import blob call to capture non-queue related wait time
- added to `test_reading_file_gets_file_from_hg` in `integration.stats_test.HgBackingStoreStatsTest` to test import blob logging in addition to the get blob loging
(not yet done for importing trees, will do in next diff)
Reviewed By: chadaustin
Differential Revision: D20201215
fbshipit-source-id: c89281fe7d3d6e89d111ac8cce9014adff44ac40
Summary:
Context: https://fb.workplace.com/groups/rust.language/permalink/3338940432821215/
This codemod replaces *all* dependencies on `//common/rust/renamed:futures-preview` with `fbsource//third-party/rust:futures-preview` and their uses in Rust code from `futures_preview::` to `futures::`.
This does not introduce any collisions with `futures::` meaning 0.1 futures because D20168958 previously renamed all of those to `futures_old::` in crates that depend on *both* 0.1 and 0.3 futures.
Codemod performed by:
```
rg \
--files-with-matches \
--type-add buck:TARGETS \
--type buck \
--glob '!/experimental' \
--regexp '(_|\b)rust(_|\b)' \
| sed 's,TARGETS$,:,' \
| xargs \
-x \
buck query "labels(srcs, rdeps(%Ss, //common/rust/renamed:futures-preview, 1))" \
| xargs sed -i 's,\bfutures_preview::,futures::,'
rg \
--files-with-matches \
--type-add buck:TARGETS \
--type buck \
--glob '!/experimental' \
--regexp '(_|\b)rust(_|\b)' \
| xargs sed -i 's,//common/rust/renamed:futures-preview,fbsource//third-party/rust:futures-preview,'
```
Reviewed By: k21
Differential Revision: D20213432
fbshipit-source-id: 07ee643d350c5817cda1f43684d55084f8ac68a6
Summary:
While we are transitioning from tokio 0.1 to tokio 0.2 we might need to use
[tokio_compat](https://docs.rs/tokio-compat/0.1.4/tokio_compat/) crate.
Let's add a helper macro similar to fbinit::test that uses tokio_compat
runtime.
Reviewed By: farnz
Differential Revision: D20213814
fbshipit-source-id: 18976e953011c8ada1fa915686e2dcb76ea288d5
Summary:
Well, we don't have a Tokio Compat runtime in Actix. This means Tokio 0.2 code
(e.g. Tokio 0.2 timers) blows up when executed in the API Server.
How do we fix this? By not running Mononoke code on Actix's runtime, and
instead running in on a Mononoke runtime we instantiated.
How do we do that? By passing a Tokio Compat Executor all the way down to the
place where Actix is about to consume our stream ... and at that point, we
spawn the stream on our runtime, and give Actix a dumb receiver that does work
when polled on a Tokio 0.1 runtime.
This feels like the end of the road for the API Server. Nothing about this is
even remotely sane, but it should take us through the API Server's eventual
demise and replacement with the Gotham-based EdenAPI Server, which runs on the
runtime of our choice (i.e. Tokio 0.2).
Reviewed By: farnz
Differential Revision: D20222294
fbshipit-source-id: 1646e35fe05b131b030e4962c8a7f68f72995035
Summary:
* Added intermediate (de)serializers for config types, so that we generate full Identity objects at config load time
* Implement FromStr for Identity
* Compare configured identities to presented identities in ratelimit middleware in order to decide whether or not to apply the limit
Reviewed By: krallin
Differential Revision: D20139308
fbshipit-source-id: 340c300db549575eb6d06efcbe437c0b1db4927b
Summary: We should support logging tags as well. I pass this along as a set until json construction because we do not want to have repeat values since tags are expected to be sets
Reviewed By: chadaustin
Differential Revision: D20199632
fbshipit-source-id: 2b5c94f1747a9b30d7a97b605abfd0e39928464c
Summary:
Usually we have only one repo, but in case of xrepo_commit_lookup we actually
have two. It's nice to know which permission failed
Reviewed By: krallin
Differential Revision: D20221509
fbshipit-source-id: ee98845767e72f99027ba18a8c5b374cb6f9f3ab
