Summary: Updated find_eden to find the Eden clone on a Windows system. On Windows we don't use symlinks, which make the logic different than on POSIX implementation.
Reviewed By: simpkins
Differential Revision: D19953934
fbshipit-source-id: bfbc112c3ccc48735ec6590746d8275cc9850796
Summary:
In D20130406 I updated the CLI to call `getDaemonInfo()` to check on the
server status. However, some very old EdenFS instances do not have have this
method. These instances should all be gone shortly, but for now update the
code to handle the unknown method error and fall back to calling `getPid()`
and `getStatus()` separately.
I implemented this in our `EdenClient` wrapper class, similar to our existing
wrapper for `getPid()`.
Reviewed By: fanzeyi
Differential Revision: D20212518
fbshipit-source-id: 9d48bdd26822802a7e9776128c5567436d4bb445
Summary:
Update the import statements so that autodeps works on the `eden/py`
directories.
Reviewed By: fanzeyi
Differential Revision: D20212519
fbshipit-source-id: 37ccabf14dc0dbfe998664260ae9b83c9136ad63
Summary:
Update process_finder.py to look for a process's Eden state directory by
looking through its open FDs to find the EdenFS lock file, if it can't find
the state directory from the command line arguments.
At the moment we almost always invoke EdenFS with an explicit `--edenDir`
argument, but this code will allow this code to work even if we remove that in
the future.
Reviewed By: wez
Differential Revision: D20178484
fbshipit-source-id: 361b78f4a2566b8c09ce02fb21c46233d7e2546b
Summary:
Update the `FakeProcessFinder.add_edenfs()` function to also add a fake
privhelper process in addition to the main edenfs process. This allows the
tests to more accurately simulate the normal edenfs behavior.
Reviewed By: wez
Differential Revision: D20178482
fbshipit-source-id: edc70ade1b61929b37f13ece77757c7c35aa4eec
Summary:
Update the code in `process_finder.py` to return EdenFS processes owned by all
users. We now report the `uid` as a field in the returned process info, so
that callers can filter the results based on user ID if they want. This
allows callers more flexibility when finding processes.
This also updates the `FakeProcessFinder` test utility code to support
providing fake UIDs to allow testing this behavior.
Reviewed By: wez
Differential Revision: D20178490
fbshipit-source-id: 6b76e1109e4835b167c80688fd3ace50f7986a22
Summary:
Move the code to find rogue EdenFS processes out of the generic
`process_finder` module and into the `check_rogue_edenfs` module that is
specific to the `eden doctor` checks.
The `ProcessFinder` class now exposes a `get_edenfs_processes()` API instead
of `find_rogue_pids()`, which makes it more generically usable outside of just
the doctor code.
Reviewed By: wez
Differential Revision: D20178486
fbshipit-source-id: e289f1673a5d4a666e9d54e8f58f4f00bdde94b7
Summary: - added logging only around the import tree call to capture non-queue related wait time
Reviewed By: chadaustin, fanzeyi
Differential Revision: D20207472
fbshipit-source-id: d88bb34ce224a26ff2be100d7789ddeff608006d
Summary:
- added logging only around the import blob call to capture non-queue related wait time
- added to `test_reading_file_gets_file_from_hg` in `integration.stats_test.HgBackingStoreStatsTest` to test import blob logging in addition to the get blob loging
(not yet done for importing trees, will do in next diff)
Reviewed By: chadaustin
Differential Revision: D20201215
fbshipit-source-id: c89281fe7d3d6e89d111ac8cce9014adff44ac40
Summary:
Context: https://fb.workplace.com/groups/rust.language/permalink/3338940432821215/
This codemod replaces *all* dependencies on `//common/rust/renamed:futures-preview` with `fbsource//third-party/rust:futures-preview` and their uses in Rust code from `futures_preview::` to `futures::`.
This does not introduce any collisions with `futures::` meaning 0.1 futures because D20168958 previously renamed all of those to `futures_old::` in crates that depend on *both* 0.1 and 0.3 futures.
Codemod performed by:
```
rg \
--files-with-matches \
--type-add buck:TARGETS \
--type buck \
--glob '!/experimental' \
--regexp '(_|\b)rust(_|\b)' \
| sed 's,TARGETS$,:,' \
| xargs \
-x \
buck query "labels(srcs, rdeps(%Ss, //common/rust/renamed:futures-preview, 1))" \
| xargs sed -i 's,\bfutures_preview::,futures::,'
rg \
--files-with-matches \
--type-add buck:TARGETS \
--type buck \
--glob '!/experimental' \
--regexp '(_|\b)rust(_|\b)' \
| xargs sed -i 's,//common/rust/renamed:futures-preview,fbsource//third-party/rust:futures-preview,'
```
Reviewed By: k21
Differential Revision: D20213432
fbshipit-source-id: 07ee643d350c5817cda1f43684d55084f8ac68a6
Summary:
While we are transitioning from tokio 0.1 to tokio 0.2 we might need to use
[tokio_compat](https://docs.rs/tokio-compat/0.1.4/tokio_compat/) crate.
Let's add a helper macro similar to fbinit::test that uses tokio_compat
runtime.
Reviewed By: farnz
Differential Revision: D20213814
fbshipit-source-id: 18976e953011c8ada1fa915686e2dcb76ea288d5
Summary:
Well, we don't have a Tokio Compat runtime in Actix. This means Tokio 0.2 code
(e.g. Tokio 0.2 timers) blows up when executed in the API Server.
How do we fix this? By not running Mononoke code on Actix's runtime, and
instead running in on a Mononoke runtime we instantiated.
How do we do that? By passing a Tokio Compat Executor all the way down to the
place where Actix is about to consume our stream ... and at that point, we
spawn the stream on our runtime, and give Actix a dumb receiver that does work
when polled on a Tokio 0.1 runtime.
This feels like the end of the road for the API Server. Nothing about this is
even remotely sane, but it should take us through the API Server's eventual
demise and replacement with the Gotham-based EdenAPI Server, which runs on the
runtime of our choice (i.e. Tokio 0.2).
Reviewed By: farnz
Differential Revision: D20222294
fbshipit-source-id: 1646e35fe05b131b030e4962c8a7f68f72995035
Summary:
* Added intermediate (de)serializers for config types, so that we generate full Identity objects at config load time
* Implement FromStr for Identity
* Compare configured identities to presented identities in ratelimit middleware in order to decide whether or not to apply the limit
Reviewed By: krallin
Differential Revision: D20139308
fbshipit-source-id: 340c300db549575eb6d06efcbe437c0b1db4927b
Summary: We should support logging tags as well. I pass this along as a set until json construction because we do not want to have repeat values since tags are expected to be sets
Reviewed By: chadaustin
Differential Revision: D20199632
fbshipit-source-id: 2b5c94f1747a9b30d7a97b605abfd0e39928464c
Summary:
Usually we have only one repo, but in case of xrepo_commit_lookup we actually
have two. It's nice to know which permission failed
Reviewed By: krallin
Differential Revision: D20221509
fbshipit-source-id: ee98845767e72f99027ba18a8c5b374cb6f9f3ab
Summary: publish per-node-type progrss stats so we can correlate storage access/load to type of node traversed
Reviewed By: farnz
Differential Revision: D20181064
fbshipit-source-id: c741b526c50e86a3eee105fab57fd7bc3ecc063b
Summary: Add tests for existing default block casefolding_check behaviour, plus test demonstrating problem with casefolding_check=false
Reviewed By: farnz
Differential Revision: D20192412
fbshipit-source-id: 1aea0fc5581e0c44388a4224ca693698731d3cd5
Summary:
In targets that depend on *both* 0.1 and 0.3 futures, this codemod renames the 0.1 dependency to be exposed as futures_old::. This is in preparation for flipping the 0.3 dependencies from futures_preview:: to plain futures::.
rs changes performed by:
```
rg \
--files-with-matches \
--type-add buck:TARGETS \
--type buck \
--glob '!/experimental' \
--regexp '(_|\b)rust(_|\b)' \
| sed 's,TARGETS$,:,' \
| xargs \
-x \
buck query "labels(srcs,
rdeps(%Ss, fbsource//third-party/rust:futures-old, 1)
intersect
rdeps(%Ss, //common/rust/renamed:futures-preview, 1)
)" \
| xargs sed -i 's/\bfutures::/futures_old::/'
```
Reviewed By: jsgf
Differential Revision: D20168958
fbshipit-source-id: d2c099f9170c427e542975bc22fd96138a7725b0
Summary: Recently there are increased reports on EdenFS's backing repo stuck in interrupted transaction state, and the user has to manually run `hg recover` in their backing repo to fix the problem. This diff teaches `eden doctor` to automatically run that command for the users.
Reviewed By: simpkins
Differential Revision: D20109567
fbshipit-source-id: a7427834e98425be388741c7f214b9d7354ac44e
Summary:
Enable pyre-strict type checking for `process_finder.py`, and split it into
its own library.
Reviewed By: genevievehelsel
Differential Revision: D20178483
fbshipit-source-id: e6c62ca5d84c7b7e599ae00fb51df6f7e4c55a65
Summary:
A couple places in the CLI code (mostly used by `eden doctor`) were checking
`sys.platform` to tell if we were on Linux. Unfortunately these checks both
expected the value `linux2`. However, since Python 3.3 `sys.platform` is just
`linux` on Linux, and not `linux2`. This meant we were always hitting the
non-Linux code paths and skipping these checks.
This updates the code to check `platform.system()`. Based on the
documentation it sounds like this is intended to give a bit more consistent
behavior across different platforms and OS versions.
Reviewed By: genevievehelsel
Differential Revision: D20178488
fbshipit-source-id: c908d5133a9c41e6a239a8893742d03f6c08527c
Summary:
Call `folly::setThreadName()` in the privhelper process when it starts. This
changes the command name reported in `/proc/PID/comm` and in `ps`
The process name is limited to 15 bytes, so this shows up as `edenfs_privhelp`
Reviewed By: fanzeyi
Differential Revision: D20199409
fbshipit-source-id: a5349bfab9230174aaa99c87f0db73fe31659186
Summary: One small step towards typing
Reviewed By: thatch
Differential Revision: D20090620
fbshipit-source-id: 811bb54159ab91e5560d115c20373eaf6542b2f9
Summary:
Small cleanup that removes a bunch of duplicate code.
That should make it easier to add other types of derived data to the warmer
Reviewed By: krallin
Differential Revision: D20193169
fbshipit-source-id: 437fe7981d8a71164dc9edfcc423e8c41cbe0967
Summary:
Add a a new `hg` module to the `mononoke_api` crate that provides a `HgRepoContext` type, which can be used to query the repo for data in Mercurial-specific formats. This will be used in the EdenAPI server.
Initially, the `HgRepoContext`'s functionality is limited to just getting the content of individual files. It will be expanded to support querying more things in later diffs.
Reviewed By: markbt
Differential Revision: D20117038
fbshipit-source-id: 23dd0c727b9e3d80bd6dc873804e41c7772f3146
Summary:
This updates our middleware stack and introduces two new pieces of functinality:
- Middleware can now be async.
- Middleware can now preempt requests and dispatch a response.
The underlying motivation for this is to allow implementing Mononoke LFS's rate
limiting middleware in our existing middleware stack.
Reviewed By: kulshrax
Differential Revision: D20191213
fbshipit-source-id: fc1df7a14eb0bbefd965e32c1fca5557124076b5
Summary: D20121350 changed the methods for accessing file content on `FileContext` to no longer return `Stream`s. We should update the comments accordingly.
Reviewed By: ahornby
Differential Revision: D20160128
fbshipit-source-id: f5bfd7e31bc7e6db63f56b8f4fc238893aa09a90
Summary:
A bunch of files include folly/executors/GlobalExecutors.h transitively through thrift/lib/cpp2/async/Stream.h, which is going away. Explicitly include the header (and add dependency to target) in preparation for deleting Stream.h
drop-conflicts
Reviewed By: vitaut
Differential Revision: D20141838
fbshipit-source-id: 21c58cf82136287fc2d84ba5badec6b872106015
Summary:
This updates the hg_sync_job to update Globalrevs in hgsql before attempting to
sync bundles. This means that if we're syncing successfully, hg is in sync with
Mononoke, and if we fail (which should be very uncommon to begin with!), hg
might skip a little bit ahead, but that's OK.
This only makes sense when generating bundles — when doing pushrebase, hg would
be updating its own globalrevs.
Reviewed By: StanislavGlebik
Differential Revision: D20159262
fbshipit-source-id: 6736f8592682da1001c7c9c4c9444462b71913c2
Summary:
Our previous implementation of unodes had a problem with diamond merges -
essentially because p1 and p2 might have the same file but with different
content unode will always create a merge unode which can be unexpected.
(code comment in unodes/derive.rs has more info about it).
This diff fixes the problem by introducing unodes v2. This allows us to import
new repos with new unode implementation while keeping the old repos with unode
v1.
This implementation uses a heuristic which should be fast and should do the
correct thing most of the time. In some cases it might exclude some parts of
the history completely. For example:
O <- merge commit, doesn't change anything
/ \
P1 | <- modified "file.txt" to "B"
| P2 <- modified "file.txt" to "B"
\ /
ROOT <- created "file.txt" with content "A"
In that case history of "file.txt" starting from merge commit will contain only (P1, ROOT),
but it won't contain P2.
We also considered other options:
1) Move this heuristic to fastlog batch derived data. See D19973553 for more
details about why we decided not to do it.
2) Filter out parent unodes that are ancestors of other parent unodes. This should
always be correct, but it will be hard to implement, it wil be even harder to make
sure it always have good performance.
Reviewed By: krallin
Differential Revision: D19978157
fbshipit-source-id: 445ddd5629669d987e7aa88c35fecf0b34a40da0
Summary: I'd like to log all derivations to a single place so that's it's easier to understand what was derived and where
Reviewed By: aslpavel
Differential Revision: D20140004
fbshipit-source-id: 305ea533031a04ff95995a6fe2a6e57e95a87026
Summary: Log the source node when validating so that we can more quickly reproduce any issues in a single step via the --walk-root option, rather than needing to run the entire walk again.
Differential Revision: D20098200
fbshipit-source-id: 6b0d7d151c97f25080953d6c0fbf431dc2cec6a8
Summary:
Also patch aho-corasick to fix the issue.
The issue was introduced by [an optimization path](063ca0d253) added in aho-corasick 0.7 series (used by globset 0.4.3).
aho-corasick 0.6.x (globset 0.4.2) are not affected.
The next aho-corasick release (0.7.9) contains the fix.
See https://github.com/BurntSushi/aho-corasick/issues/53 for more context.
Reported by: yns88
Reviewed By: DurhamG
Differential Revision: D20125697
fbshipit-source-id: 592375b43d7ee494bb3e916a1cb11c18f9ebe425
Summary:
`parentfunc` is only needed when adding new nodes to the DAG.
Move it to `addheads` methods instead.
Reviewed By: sfilipco
Differential Revision: D20155398
fbshipit-source-id: 0bddd5f46e84c44891928b9f598a38206917aecb
Summary:
Migrate away from some uses of revision numbers.
Some dead code in discovery.py is removed.
I also fixed some test issues when I run tests locally.
Reviewed By: sfilipco
Differential Revision: D20155399
fbshipit-source-id: bfdcb57f06374f9f27be51b0980652ef50a2c8e0
Summary:
`hiddenoverride` is a hacky implementation that preserves part of another hacky
`inhibit` extension. With our modern setup (inhibit or narrow-heads),
`hiddenoverride` is less useful. Therefore just remove it.
Reviewed By: sfilipco
Differential Revision: D20148011
fbshipit-source-id: f4a5f05b67ae6f315e9b07d50ef03018d6d05df5
Summary:
This makes it so that DAG calculations in NameDag are all using commit hashes.
The `id2node`, `node2id` APIs are still using integer ids, and hopefully their
usage can eventually be removed.
Reviewed By: sfilipco
Differential Revision: D20020527
fbshipit-source-id: ee32b1ccacabd5174ff1556e426b5ed32d2b8507
Summary:
This exposes the NameSet type to the Python world.
The code is similar to the SpanSet wrapper that exists in pydag.
Reviewed By: sfilipco
Differential Revision: D20020521
fbshipit-source-id: 840e009eadca7154f11ca61561da4c48022088f6
Summary: This makes it possible to use NameIter in py_class.
Reviewed By: sfilipco
Differential Revision: D20020529
fbshipit-source-id: b9147b7dccb38d18d8361b420507fcbe97e01351
Summary:
Mercurial has a special case that b'\0' * 20 maps to rev -1 and means
"an empty commit". This cannot be cleanly supported by the zstore commit data,
since sha1("") is not '\0' * 20 and zstore does not allow faked SHA1 keys.
Therefore let's add the special case in the bindings layer. It's possible to
do this check in Python, but that'll be slower.
Reviewed By: sfilipco
Differential Revision: D20020520
fbshipit-source-id: 0686832666646f2e201035992e3951b47c32eb5a
Summary: Use the new NameDag as the backing structure and expose its APIs.
Reviewed By: sfilipco
Differential Revision: D20020528
fbshipit-source-id: ccb49e1a5e757bd35a3f71cfb54ceccfb544664e
Summary: This will be used by commit hash prefix lookup.
Reviewed By: sfilipco
Differential Revision: D20020523
fbshipit-source-id: f2905ddf63098704b08dad8eb48272c3ffba7e25
Summary: Export common types at the top-level of the crate so it's easier to use.
Reviewed By: sfilipco
Differential Revision: D20020526
fbshipit-source-id: e9a0a8bc3cc91f81d0bc74e7530cd4613fc1dd61
Summary: Those just delegate to IdDag for the actual calculation.
Reviewed By: sfilipco
Differential Revision: D20020522
fbshipit-source-id: 272828c520097c993ab50dac6ecc94dc370c8e8b
