Summary: Add a method to `HgFileContext` to stream the history of the file. Will be used to support EdenAPI history requests.
Reviewed By: krallin
Differential Revision: D20211779
fbshipit-source-id: 49e8c235468d18b23976e64a9205cbcc86a7a1b4
Summary: Add an 'HgTreeContext' struct to the 'hg' module to allow querying for tree data in Mercurial-specific formats. This initial implementation's primary purpose is to enable getting the content of tree nodes in a format that can be written directly to Mercurial's storage.
Reviewed By: krallin
Differential Revision: D20159958
fbshipit-source-id: d229aee4d6c7d9ef45297c18de6e393d2a2dc83f
Summary: I was looking in the `edenfs_events` table and saw that sandcastle was logging to this table. Rice was able to identify that the reason was because the integration tests were logging. So if we're on running integration tests, we should return a `NullTelemetryLogger`. The daemon currently does not log on sandcastle AFAIK.
Reviewed By: simpkins
Differential Revision: D20203556
fbshipit-source-id: e09175347631478cb366d4fa2c6092d976504dd8
Summary: `buck run edenfsctl -- du --clean` would help reduce the space used by the storage engine.
Reviewed By: chadaustin
Differential Revision: D20200616
fbshipit-source-id: 6ffa588fc71660a6a80d81aef7d58dda08932374
Summary:
Add a `get_build_info()` method to the `EdenFSProcess` objects returned by
the `process_finder` module. This returns information about the process
version and build time.
Reviewed By: wez
Differential Revision: D20178487
fbshipit-source-id: b1eb41de9184ca59dc1e90d0a92ff1cbc89a6b77
Summary:
Store member variables in a local variable so that Pyre will allow unwrapping
it from an `Optional` type. Pyre refuses to allow member variables to be
extracted from `Optional` since other functions called indirectly could modify
them.
Reviewed By: fanzeyi
Differential Revision: D20212162
fbshipit-source-id: 95655b73b5e469688f48d402c0b587928cbb0a35
Summary:
This makes it clear that `log` is a math concept, not an append-only file like
`Log`.
Reviewed By: DurhamG
Differential Revision: D20149376
fbshipit-source-id: 67d2e9584b15f48759ca9b6dfce4279a5b1365a0
Summary: This makes it friendly to Python 2.
Reviewed By: sfilipco
Differential Revision: D20162233
fbshipit-source-id: 5beb7a0f52159afc454332ff6e37e13087177cc0
Summary:
When I run `hg doctor` in my www checkout it fails the assertion check of the
first line of visibleheads is "v1". Make it graceful so doctor can check and
fix other components.
Reviewed By: DurhamG
Differential Revision: D20147969
fbshipit-source-id: 6aee2cab962fcd0ef06a0611d288021e86621249
Summary: Updated find_eden to find the Eden clone on a Windows system. On Windows we don't use symlinks, which make the logic different than on POSIX implementation.
Reviewed By: simpkins
Differential Revision: D19953934
fbshipit-source-id: bfbc112c3ccc48735ec6590746d8275cc9850796
Summary:
In D20130406 I updated the CLI to call `getDaemonInfo()` to check on the
server status. However, some very old EdenFS instances do not have have this
method. These instances should all be gone shortly, but for now update the
code to handle the unknown method error and fall back to calling `getPid()`
and `getStatus()` separately.
I implemented this in our `EdenClient` wrapper class, similar to our existing
wrapper for `getPid()`.
Reviewed By: fanzeyi
Differential Revision: D20212518
fbshipit-source-id: 9d48bdd26822802a7e9776128c5567436d4bb445
Summary:
Update the import statements so that autodeps works on the `eden/py`
directories.
Reviewed By: fanzeyi
Differential Revision: D20212519
fbshipit-source-id: 37ccabf14dc0dbfe998664260ae9b83c9136ad63
Summary:
Update process_finder.py to look for a process's Eden state directory by
looking through its open FDs to find the EdenFS lock file, if it can't find
the state directory from the command line arguments.
At the moment we almost always invoke EdenFS with an explicit `--edenDir`
argument, but this code will allow this code to work even if we remove that in
the future.
Reviewed By: wez
Differential Revision: D20178484
fbshipit-source-id: 361b78f4a2566b8c09ce02fb21c46233d7e2546b
Summary:
Update the `FakeProcessFinder.add_edenfs()` function to also add a fake
privhelper process in addition to the main edenfs process. This allows the
tests to more accurately simulate the normal edenfs behavior.
Reviewed By: wez
Differential Revision: D20178482
fbshipit-source-id: edc70ade1b61929b37f13ece77757c7c35aa4eec
Summary:
Update the code in `process_finder.py` to return EdenFS processes owned by all
users. We now report the `uid` as a field in the returned process info, so
that callers can filter the results based on user ID if they want. This
allows callers more flexibility when finding processes.
This also updates the `FakeProcessFinder` test utility code to support
providing fake UIDs to allow testing this behavior.
Reviewed By: wez
Differential Revision: D20178490
fbshipit-source-id: 6b76e1109e4835b167c80688fd3ace50f7986a22
Summary:
Move the code to find rogue EdenFS processes out of the generic
`process_finder` module and into the `check_rogue_edenfs` module that is
specific to the `eden doctor` checks.
The `ProcessFinder` class now exposes a `get_edenfs_processes()` API instead
of `find_rogue_pids()`, which makes it more generically usable outside of just
the doctor code.
Reviewed By: wez
Differential Revision: D20178486
fbshipit-source-id: e289f1673a5d4a666e9d54e8f58f4f00bdde94b7
Summary: - added logging only around the import tree call to capture non-queue related wait time
Reviewed By: chadaustin, fanzeyi
Differential Revision: D20207472
fbshipit-source-id: d88bb34ce224a26ff2be100d7789ddeff608006d
Summary:
- added logging only around the import blob call to capture non-queue related wait time
- added to `test_reading_file_gets_file_from_hg` in `integration.stats_test.HgBackingStoreStatsTest` to test import blob logging in addition to the get blob loging
(not yet done for importing trees, will do in next diff)
Reviewed By: chadaustin
Differential Revision: D20201215
fbshipit-source-id: c89281fe7d3d6e89d111ac8cce9014adff44ac40
Summary:
Context: https://fb.workplace.com/groups/rust.language/permalink/3338940432821215/
This codemod replaces *all* dependencies on `//common/rust/renamed:futures-preview` with `fbsource//third-party/rust:futures-preview` and their uses in Rust code from `futures_preview::` to `futures::`.
This does not introduce any collisions with `futures::` meaning 0.1 futures because D20168958 previously renamed all of those to `futures_old::` in crates that depend on *both* 0.1 and 0.3 futures.
Codemod performed by:
```
rg \
--files-with-matches \
--type-add buck:TARGETS \
--type buck \
--glob '!/experimental' \
--regexp '(_|\b)rust(_|\b)' \
| sed 's,TARGETS$,:,' \
| xargs \
-x \
buck query "labels(srcs, rdeps(%Ss, //common/rust/renamed:futures-preview, 1))" \
| xargs sed -i 's,\bfutures_preview::,futures::,'
rg \
--files-with-matches \
--type-add buck:TARGETS \
--type buck \
--glob '!/experimental' \
--regexp '(_|\b)rust(_|\b)' \
| xargs sed -i 's,//common/rust/renamed:futures-preview,fbsource//third-party/rust:futures-preview,'
```
Reviewed By: k21
Differential Revision: D20213432
fbshipit-source-id: 07ee643d350c5817cda1f43684d55084f8ac68a6
Summary:
While we are transitioning from tokio 0.1 to tokio 0.2 we might need to use
[tokio_compat](https://docs.rs/tokio-compat/0.1.4/tokio_compat/) crate.
Let's add a helper macro similar to fbinit::test that uses tokio_compat
runtime.
Reviewed By: farnz
Differential Revision: D20213814
fbshipit-source-id: 18976e953011c8ada1fa915686e2dcb76ea288d5
Summary:
Well, we don't have a Tokio Compat runtime in Actix. This means Tokio 0.2 code
(e.g. Tokio 0.2 timers) blows up when executed in the API Server.
How do we fix this? By not running Mononoke code on Actix's runtime, and
instead running in on a Mononoke runtime we instantiated.
How do we do that? By passing a Tokio Compat Executor all the way down to the
place where Actix is about to consume our stream ... and at that point, we
spawn the stream on our runtime, and give Actix a dumb receiver that does work
when polled on a Tokio 0.1 runtime.
This feels like the end of the road for the API Server. Nothing about this is
even remotely sane, but it should take us through the API Server's eventual
demise and replacement with the Gotham-based EdenAPI Server, which runs on the
runtime of our choice (i.e. Tokio 0.2).
Reviewed By: farnz
Differential Revision: D20222294
fbshipit-source-id: 1646e35fe05b131b030e4962c8a7f68f72995035
Summary:
* Added intermediate (de)serializers for config types, so that we generate full Identity objects at config load time
* Implement FromStr for Identity
* Compare configured identities to presented identities in ratelimit middleware in order to decide whether or not to apply the limit
Reviewed By: krallin
Differential Revision: D20139308
fbshipit-source-id: 340c300db549575eb6d06efcbe437c0b1db4927b
Summary: We should support logging tags as well. I pass this along as a set until json construction because we do not want to have repeat values since tags are expected to be sets
Reviewed By: chadaustin
Differential Revision: D20199632
fbshipit-source-id: 2b5c94f1747a9b30d7a97b605abfd0e39928464c
Summary:
Usually we have only one repo, but in case of xrepo_commit_lookup we actually
have two. It's nice to know which permission failed
Reviewed By: krallin
Differential Revision: D20221509
fbshipit-source-id: ee98845767e72f99027ba18a8c5b374cb6f9f3ab
Summary: publish per-node-type progrss stats so we can correlate storage access/load to type of node traversed
Reviewed By: farnz
Differential Revision: D20181064
fbshipit-source-id: c741b526c50e86a3eee105fab57fd7bc3ecc063b
Summary: Add tests for existing default block casefolding_check behaviour, plus test demonstrating problem with casefolding_check=false
Reviewed By: farnz
Differential Revision: D20192412
fbshipit-source-id: 1aea0fc5581e0c44388a4224ca693698731d3cd5
Summary:
In targets that depend on *both* 0.1 and 0.3 futures, this codemod renames the 0.1 dependency to be exposed as futures_old::. This is in preparation for flipping the 0.3 dependencies from futures_preview:: to plain futures::.
rs changes performed by:
```
rg \
--files-with-matches \
--type-add buck:TARGETS \
--type buck \
--glob '!/experimental' \
--regexp '(_|\b)rust(_|\b)' \
| sed 's,TARGETS$,:,' \
| xargs \
-x \
buck query "labels(srcs,
rdeps(%Ss, fbsource//third-party/rust:futures-old, 1)
intersect
rdeps(%Ss, //common/rust/renamed:futures-preview, 1)
)" \
| xargs sed -i 's/\bfutures::/futures_old::/'
```
Reviewed By: jsgf
Differential Revision: D20168958
fbshipit-source-id: d2c099f9170c427e542975bc22fd96138a7725b0
Summary: Recently there are increased reports on EdenFS's backing repo stuck in interrupted transaction state, and the user has to manually run `hg recover` in their backing repo to fix the problem. This diff teaches `eden doctor` to automatically run that command for the users.
Reviewed By: simpkins
Differential Revision: D20109567
fbshipit-source-id: a7427834e98425be388741c7f214b9d7354ac44e
Summary:
Enable pyre-strict type checking for `process_finder.py`, and split it into
its own library.
Reviewed By: genevievehelsel
Differential Revision: D20178483
fbshipit-source-id: e6c62ca5d84c7b7e599ae00fb51df6f7e4c55a65
Summary:
A couple places in the CLI code (mostly used by `eden doctor`) were checking
`sys.platform` to tell if we were on Linux. Unfortunately these checks both
expected the value `linux2`. However, since Python 3.3 `sys.platform` is just
`linux` on Linux, and not `linux2`. This meant we were always hitting the
non-Linux code paths and skipping these checks.
This updates the code to check `platform.system()`. Based on the
documentation it sounds like this is intended to give a bit more consistent
behavior across different platforms and OS versions.
Reviewed By: genevievehelsel
Differential Revision: D20178488
fbshipit-source-id: c908d5133a9c41e6a239a8893742d03f6c08527c
Summary:
Call `folly::setThreadName()` in the privhelper process when it starts. This
changes the command name reported in `/proc/PID/comm` and in `ps`
The process name is limited to 15 bytes, so this shows up as `edenfs_privhelp`
Reviewed By: fanzeyi
Differential Revision: D20199409
fbshipit-source-id: a5349bfab9230174aaa99c87f0db73fe31659186
Summary: One small step towards typing
Reviewed By: thatch
Differential Revision: D20090620
fbshipit-source-id: 811bb54159ab91e5560d115c20373eaf6542b2f9
Summary:
Small cleanup that removes a bunch of duplicate code.
That should make it easier to add other types of derived data to the warmer
Reviewed By: krallin
Differential Revision: D20193169
fbshipit-source-id: 437fe7981d8a71164dc9edfcc423e8c41cbe0967
Summary:
Add a a new `hg` module to the `mononoke_api` crate that provides a `HgRepoContext` type, which can be used to query the repo for data in Mercurial-specific formats. This will be used in the EdenAPI server.
Initially, the `HgRepoContext`'s functionality is limited to just getting the content of individual files. It will be expanded to support querying more things in later diffs.
Reviewed By: markbt
Differential Revision: D20117038
fbshipit-source-id: 23dd0c727b9e3d80bd6dc873804e41c7772f3146
Summary:
This updates our middleware stack and introduces two new pieces of functinality:
- Middleware can now be async.
- Middleware can now preempt requests and dispatch a response.
The underlying motivation for this is to allow implementing Mononoke LFS's rate
limiting middleware in our existing middleware stack.
Reviewed By: kulshrax
Differential Revision: D20191213
fbshipit-source-id: fc1df7a14eb0bbefd965e32c1fca5557124076b5
Summary: D20121350 changed the methods for accessing file content on `FileContext` to no longer return `Stream`s. We should update the comments accordingly.
Reviewed By: ahornby
Differential Revision: D20160128
fbshipit-source-id: f5bfd7e31bc7e6db63f56b8f4fc238893aa09a90
Summary:
A bunch of files include folly/executors/GlobalExecutors.h transitively through thrift/lib/cpp2/async/Stream.h, which is going away. Explicitly include the header (and add dependency to target) in preparation for deleting Stream.h
drop-conflicts
Reviewed By: vitaut
Differential Revision: D20141838
fbshipit-source-id: 21c58cf82136287fc2d84ba5badec6b872106015
Summary:
This updates the hg_sync_job to update Globalrevs in hgsql before attempting to
sync bundles. This means that if we're syncing successfully, hg is in sync with
Mononoke, and if we fail (which should be very uncommon to begin with!), hg
might skip a little bit ahead, but that's OK.
This only makes sense when generating bundles — when doing pushrebase, hg would
be updating its own globalrevs.
Reviewed By: StanislavGlebik
Differential Revision: D20159262
fbshipit-source-id: 6736f8592682da1001c7c9c4c9444462b71913c2
