Commit Graph

7817 Commits

Author SHA1 Message Date
Jun Wu
f04e0fa60e namedag: define an abstract NameDag struct
Summary: It's not yet abstract. But one step closer.

Reviewed By: sfilipco

Differential Revision: D24399510

fbshipit-source-id: 32969733babd41b221170ee440f5d7ced1f7490a
2020-10-20 15:19:30 -07:00
Jun Wu
c49caf64b4 dag: drop snapshot_map from NameDag
Summary: The `snapshot` field can be used instead.

Reviewed By: sfilipco

Differential Revision: D24399507

fbshipit-source-id: 67de20d897b8b763f724f3ccbd46618dec7911b9
2020-10-20 15:19:30 -07:00
Jun Wu
465ac6c5df dag: drop IdMapEq trait
Summary:
The trait requires an `IdMap` snapshot to be locally ready. That's not easy for
all possible implementations. Drop it to simplify things.

Reviewed By: sfilipco

Differential Revision: D24399501

fbshipit-source-id: 4d85f77c99208cda30b2a543a0bb5b295f49a65c
2020-10-20 15:19:30 -07:00
Jun Wu
5f00c7984b dag: unify prepare_filesystem_sync
Summary: There were 2 prepare_filesystem_sync. Unify them into one implementation.

Reviewed By: sfilipco

Differential Revision: D24399513

fbshipit-source-id: 80d009c33b7f23dc2c4225da6fd0fb09589ba061
2020-10-20 15:19:30 -07:00
Jun Wu
4a7ffc4bd7 dag: use Locked for SyncableIdMap
Summary: Simplifies some code.

Reviewed By: sfilipco

Differential Revision: D24399500

fbshipit-source-id: a1317149da066617c4060b7efdae5234e5bd7262
2020-10-20 15:19:30 -07:00
Jun Wu
282c6800cb dag: use Locked for SyncableIdDag
Reviewed By: sfilipco

Differential Revision: D24399506

fbshipit-source-id: 91cfa176b8cfeca3f96dfeb211bf9d46a3d95bd5
2020-10-20 15:19:30 -07:00
Jun Wu
5c7b169e0e dag: add Locked type
Summary: More general purposed type for Syncable{IdDag,IdMap}.

Reviewed By: sfilipco

Differential Revision: D24399502

fbshipit-source-id: 0599db6dd07fe3d430458f86a33a9144d850fca1
2020-10-20 15:19:29 -07:00
Jun Wu
77790e9f49 dag: move non-master write APIs to IdMapWrite trait
Summary: This makes it more generic.

Reviewed By: sfilipco

Differential Revision: D24399493

fbshipit-source-id: 8a1d0a13dd29989b17fe3ef1497b10b6fa0629d6
2020-10-20 15:19:29 -07:00
Jun Wu
b249950984 dag: impl Persist on IdMaps
Summary: IdMap fits the Persist trait.

Reviewed By: sfilipco

Differential Revision: D24399494

fbshipit-source-id: 97b84d155f4b9bb3006bfad116defa4fca6330d6
2020-10-20 15:19:29 -07:00
Jun Wu
625b8ab4d5 dag: move IdMap impls to separate files
Summary: Similar to IdDag change, move impls to separate files.

Reviewed By: sfilipco

Differential Revision: D24399508

fbshipit-source-id: 575b6e7194677b67b6755b0a30ae7d014d498b10
2020-10-20 15:19:29 -07:00
Jun Wu
34aa41f24f dag: iddagstore::GetLock -> ops::Persist
Summary:
The lock, reload, mutate, persist pattern is general. It can be used for IdMap
too.

Reviewed By: sfilipco

Differential Revision: D24399512

fbshipit-source-id: d25e51ba735061ca101101d75aff95deb88b1d36
2020-10-20 15:19:29 -07:00
Jun Wu
f01dca1995 dag: drop build_segments_persistent APIs
Summary:
Now `build_segments_persistent` and `build_segments_volatile` are the same.
Just keep one of them.

Reviewed By: sfilipco

Differential Revision: D24399511

fbshipit-source-id: a9f1ac920cdf5b448bd99bf9b6d4ca4160ba0304
2020-10-20 15:19:29 -07:00
Jun Wu
5fad63b010 dag: drop last high level segment by default
Summary:
Previously, we keep the last high level segment per level in memory, and
drop it on disk. When we cross the memory / disk boundary, we had to
maintain such properties carefully. That was needed because some DAG
algorithms rely on complete high level segments.

Now that no DAG algorithms depend on such properties, let's just drop
the logic adding the last segment back to simplify the code.

This removes the need of building segments after open() and sync().

Reviewed By: sfilipco

Differential Revision: D24399515

fbshipit-source-id: 4c640d9aa03c050fcd97f70ee386e32d3a8ee26d
2020-10-20 15:19:29 -07:00
Jun Wu
496724e45e dag: make children work with missing high-level segments
Summary:
This makes the algorithm a bit more robust. Now none of the DAG algorithms
depend on high-level segments are complete and cover all low-level segments.

This also removes constraints. For example, SyncableIdDag can now just
deref() to the normal IdDag for queries without worrying about correctness.

Reviewed By: sfilipco

Differential Revision: D24399503

fbshipit-source-id: e6a91010cff82264cf423e2f24dee1d372822ef6
2020-10-20 15:19:29 -07:00
Jun Wu
f290afe421 dag: remove {range,descendants}_old algorithms
Summary:
They depend on high-level segments covering low-level segments, which
adds extra complexities. Remove them to simplify logic.

Reviewed By: sfilipco

Differential Revision: D24399509

fbshipit-source-id: 56a8e06c263107d1da4d6754b884ce51e18e30bf
2020-10-20 15:19:29 -07:00
Durham Goode
c80aaa4b56 ui: disable ui.threaded by default
Summary:
Even if ui.threaded is off, we're seeing sys call interrupt issues. I
think this is because ui.threaded defaults to true, and we check it's value
before loading the repo config (which includes the dynamicconfig to turn it
off). It's already off in dynamicconfig, so let's turn it off by default.

This highlights an issue with dynamicconfig, in that it can't deploy pre-repo
configs at the moment.

As for the actual interrupted syscall, it looks to be caused by SIGCHLD signals
from child processes we spawn (possibly the dynamicconfig child process).

Reviewed By: quark-zju

Differential Revision: D24401521

fbshipit-source-id: 0372ffc7297fa6fe6a7e4f0ae69299b252c3632e
2020-10-20 12:18:44 -07:00
Chad Austin
8394946bc9 allow tracing fuse events with detailed argument rendering
Summary:
Provide an API that allows a FuseChannel TraceBus subscriber to
request population of rendered argument strings. This is not on by
default to avoid the cost in the common case that there are no
subscribers that use the field.

Reviewed By: kmancini

Differential Revision: D24036018

fbshipit-source-id: ff12252de3777606360efc75814149931637aa9b
2020-10-20 09:34:03 -07:00
Chad Austin
265ca32efe generalize strace log rendering
Summary:
Provide a standard argument renderer per HandlerEntry so it can be
used outside of the strace logger.

Reviewed By: kmancini

Differential Revision: D24035882

fbshipit-source-id: f3f1d918b1a7c937777e2c5fc0de981b5ca3286c
2020-10-20 09:34:03 -07:00
Chad Austin
f6fcff3151 move strace logging into FuseChannel
Summary:
Instead of logging in the Dispatcher, move strace logging to
FuseChannel where it can be standardized for all FUSE request types.

Reviewed By: wez

Differential Revision: D24035838

fbshipit-source-id: c84d8c27b62f9944e2d26a35a7ed7bbbeeb5bf0e
2020-10-20 09:34:03 -07:00
Stanislau Hlebik
aa743595ad mononoke: fix ordering in the error message
Summary: As we saw today it was showing entries in incorrect order

Reviewed By: ikostia

Differential Revision: D24418800

fbshipit-source-id: c926c7904338ae40b26fe08cb1ac973e384bad28
2020-10-20 07:41:48 -07:00
Stanislau Hlebik
01a41013e1 mononoke: remove noisy logging
Summary: It's very spammy

Reviewed By: ikostia

Differential Revision: D24418801

fbshipit-source-id: 35c804764e9f65a49ba73a0d983636665dae1841
2020-10-20 07:41:48 -07:00
Mark Thomas
7c8ee4f52f scs_server: add file_diff method
Summary:
Add a method to diff file contents directly, even if their paths are not known.

This provides a headerless diff, just of the content differences.

Reviewed By: ahornby

Differential Revision: D24200803

fbshipit-source-id: 32cf461d0c43a7b785ae58bec284978ff7976702
2020-10-20 05:45:39 -07:00
Mark Thomas
8945452492 mononoke_api: accept impl Into<ChangesetSpecifier> for changesets
Summary:
Rather than requiring the caller to construct a changeset specifier for known changeset
ID types, accept anything that implements `Into<ChangesetSpecifier>` and add automatic
conversions for the specifier types.

This makes use of the Mononoke API a bit more ergonomic in tests.

Reviewed By: krallin

Differential Revision: D24417892

fbshipit-source-id: ff84c833442d621d71dbadca596222a2e13d5ca1
2020-10-20 05:45:39 -07:00
Kostia Balytskyi
2da4ec1801 x-repo validation: make it work in both directions
Summary:
Prior to this diff, validation logic did not work when target repo was a large
one (although this limitation wasn't advertised). This was becase we explicitly
never apply a `Mover` to the target repo paths and therefore expect every
single target repo path to have an equivalent in the source repo. This is true
when the target repo is small, but not true when it is large.

It is pretty easy to make this logic direction-agnostic: just proactively check
whether target repo paths rewrite into nothingness or not.

Reviewed By: StanislavGlebik

Differential Revision: D24399029

fbshipit-source-id: 9ca5bb03760e662ff6756c27c0c77204abdb38de
2020-10-20 02:13:39 -07:00
Chad Austin
97fe7b4efd stop returning meaningful JournalPosition results from subscribeStreamTemporary
Summary:
Watchman, the only consumer of subscribeStreamTemporary today, does
not use the published JournalPosition values. Instead, it waits a bit,
and calls either getCurrentJournalPosition or
getFilesChangedSince. This is a good thing, because write traffic to a
filesystem can easily exceed the rate at which Watchman (or even
streaming Thrift!) can consume it, spiking memory usage and degrading
performance.

The next diff in this stack will take advantage of this clarified
communication protocol to reduce edenfs memory usage under heavy write
traffic.

Reviewed By: wez

Differential Revision: D24089811

fbshipit-source-id: 49de6be24a78934888162a577a554d50e2fff355
2020-10-19 17:33:27 -07:00
Jun Wu
0f2c7b4abe localrepo: move changelog backend selection to a separate method
Summary:
Move logic to select a changelog backend to a separate method so it can be used
by bundlerepo.

Reviewed By: DurhamG

Differential Revision: D24214463

fbshipit-source-id: 6667317e9d51076b4a5d4a07779792f27cbd6d5e
2020-10-19 17:07:31 -07:00
Jun Wu
0498f87da1 lock: detect deadlock in a same process
Summary:
This is for cases like:

    repo1 = hg.repository(path)
    repo2 = hg.repository(path)  # same path
    with repo1.lock():
        repo2.lock()  # deadlock

Previously it simply deadlocks with confusing message:

    waiting for lock held by process X  # X is the current process

With this change it will raise a ProgrammingError about deadlock.

Reviewed By: DurhamG

Differential Revision: D24214458

fbshipit-source-id: 57faddaed6d2f1d36fab067c2e1f63b3cda0a1fe
2020-10-19 17:07:30 -07:00
Jun Wu
696883188b cmdutil: rename rustdisplaygraph to displaygraph
Summary:
Replace the legacy `displaygraph` with `rustdisplaygraph`. The `edegfn`
parameter is no longer used and is dropped.

Reviewed By: DurhamG

Differential Revision: D24299592

fbshipit-source-id: 8336da1c8831b3352a950544d72cbdcf0203858d
2020-10-19 17:07:30 -07:00
Jun Wu
d394108655 graphmod: drop legacy rendering logic
Summary: Only keep the new Rust renderer.

Reviewed By: DurhamG

Differential Revision: D24299593

fbshipit-source-id: a0705ce477a5e1166a71fbadac051ae093b4598d
2020-10-19 17:07:30 -07:00
Jun Wu
2af1daa778 smartlog: remove support for legacy graph renderer
Summary: Always use the Rust graph renderer.

Reviewed By: DurhamG

Differential Revision: D24299594

fbshipit-source-id: 0f9b9688d2447ca4f05bfd50fc133e2606d2020b
2020-10-19 17:07:30 -07:00
Jun Wu
9bf6b674a6 config: use Rust graph render as default
Summary: Change the legacy Python graph render to Rust renderer.

Reviewed By: DurhamG

Differential Revision: D24317802

fbshipit-source-id: 4c3dc3a6dd02b7ebe79596a8e77f4b6b139d2e20
2020-10-19 17:07:30 -07:00
Kostia Balytskyi
65c55d0038 megarepotool: add check-push-redirection-prereqs subcommand
Summary: This command will help us understand if it is safe to enable push-redirection with a certain CommitSyncConfigVersion,

Reviewed By: StanislavGlebik

Differential Revision: D24345898

fbshipit-source-id: c2a4c034a9203025ce3534986b3bb986784ff2b1
2020-10-19 15:16:27 -07:00
Jun Wu
9743d2f06a changelog2: keep laziness of findcommonmissing
Summary:
`onlyboth` returns a lazy set for revlog backend.
`torevs` does not work well with such lazy set.
Use `torevset` instead.

Reviewed By: DurhamG

Differential Revision: D24397872

fbshipit-source-id: e7f5c6412b4120305b67103ceff1b1922a67ec7d
2020-10-19 13:17:35 -07:00
Genevieve Helsel
46c1199efb move specs out of fbsource/tools/ and into fbsource/xplat/
Reviewed By: zertosh

Differential Revision: D24287371

fbshipit-source-id: fa28caa28864ead1d724f3851778b1490bdc2026
2020-10-19 12:43:06 -07:00
Chad Austin
e146f08d9a indicate to coreutils that edenfs is a remote filesystem
Summary:
GNU `df` (and any other coreutil that relies on gnulib's ME_REMOTE
flag) detects remote filesystems with some heuristics. One of which is
whether the device type contains a colon. Since edenfs is a remote
filesystem, include a colon, so it's properly detected as such.

Reviewed By: genevievehelsel

Differential Revision: D24174015

fbshipit-source-id: 4b1f2f49c6eee6e690a6f570924274060a66eee7
2020-10-19 12:13:54 -07:00
Stanislau Hlebik
6c634f94ee mononoke: admin command to change mapping version
Summary:
We'll use this command to change the mapping version we use when doing push
redirection.

Reviewed By: ikostia

Differential Revision: D24392308

fbshipit-source-id: 4dab01c0e58a8953a0c6c84c7c166977a6baf00f
2020-10-19 09:46:29 -07:00
Stanislau Hlebik
9ec52564c7 mononoke: force users to specify mapping version in
Summary:
Previously it was just taking the current version. Instead of guessing let's
allow users to specify whatever version they want to rewrite it with.

Reviewed By: ikostia

Differential Revision: D24360918

fbshipit-source-id: d9f4c55cd66931a9f2ab7da3474d2152d77525d0
2020-10-19 09:46:29 -07:00
Stanislau Hlebik
631d1da786 mononoke: remove update_mapping method
Summary:
update_mapping method was silently setting current mapping version. This was
very error-prone and we had a replacement update_mapping_with_version. This
diff removes update_mapping completely.

Reviewed By: krallin

Differential Revision: D24360105

fbshipit-source-id: 57761f4279f75032e9d4ec88a45e5199e250247a
2020-10-19 09:46:29 -07:00
Stanislau Hlebik
5bce95a9d7 mononoke: remove copy-paste initialization of CommitSyncer
Reviewed By: krallin

Differential Revision: D24360097

fbshipit-source-id: bb4fd2de905731a2780ff9cb49f0174a3c0bbdd6
2020-10-19 09:46:29 -07:00
Pavel Aslanov
d752cfa651 batched tailing mode
Summary:
This diff add new mode of tailing based on derived data graph, it uses same functionality as backfill.
- `tail_batch_iteration` uses `bounded_traversal_dag` directly instead of leveraging `DeriveGraph::derive` so we could warm-up dependencies for each node before calling `DerivedUitls::backfill_batch_dangerous`

Reviewed By: StanislavGlebik

Differential Revision: D24306156

fbshipit-source-id: 006cb6d4df9424cd6501cb4a381b95f096e70551
2020-10-19 07:30:03 -07:00
Lukas Piatkowski
87db93687e rust/shed/hostname: rename hostname dependency to unbreak doctests (#15)
Summary:
Pull Request resolved: https://github.com/facebookexperimental/rust-shed/pull/15

With the update from rustc 1.44 to 1.47 the doctests started to break when building a crate that depends on a different crate of the same name, see this: https://github.com/rust-lang/cargo/issues/6819

This diff also removes the TravisCI build as it was broken and unused for some time.

Reviewed By: krallin

Differential Revision: D24390815

fbshipit-source-id: fa9c78ef394640afbf8c419f8276bbf339960406
2020-10-19 07:21:38 -07:00
Thomas Orozco
fde21ba026 mononoke/lfs_server: log client attempt
Summary:
I updated the client to send this earlier this week. Let's start collecting it
server side. The bottom line is this will let us identify when we reject a
client that is done retrying.

Reviewed By: HarveyHunt

Differential Revision: D24361884

fbshipit-source-id: b31e53c7dc989f1698e28b2a0bf14dc3fdbb507a
2020-10-19 05:01:57 -07:00
Jan Mazur
d7cf79cf5a print regex when rejecting a file
Summary: It will be much more convenient for end user to know why certain limit is enforced.

Reviewed By: krallin

Differential Revision: D24364473

fbshipit-source-id: ad86384f7f7a246bc3735028339cfd54a1d0254a
2020-10-19 03:27:50 -07:00
Jan Mazur
805523a1f9 adjust integration tests because of changed IpAddr Debug trait
Summary:
I think that this: 6293dca1e8
"broke" (in a sense that they need to be adjusted - run with --interactive) some of our integration tests.
In fbcode seem like it was D24363513.
This commit deleted derived Debug trait of enum IpAddr in favor of implementing it with already existing fmt::Display.
I think this change landed with rust 1.7.0 which hit fbcode recently.

Reviewed By: StanislavGlebik

Differential Revision: D24382155

fbshipit-source-id: bce5998015bfdd7c0b01a5c6ee5a661f9f00df5a
2020-10-19 01:30:28 -07:00
Chad Austin
bef06ca978 don't load hidden config files
Summary:
Chef writes root-owned temporary files in /etc/eden/config.d. If
edenfsctl attempts to read those configs at this time, it will
fail. Just skip all hidden files.

Reviewed By: kmancini

Differential Revision: D24377953

fbshipit-source-id: 765b7f096b62e6749e186adc3e90bbc7814079c8
2020-10-16 20:50:32 -07:00
Chad Austin
83dbe9d985 compute deterministic paths in debugInodeStatus
Summary:
InodeBase::getPath is not deterministic under concurrent
renames. Instead, thread the paths down through the recursion so
they're consistent.

Reviewed By: genevievehelsel

Differential Revision: D24297724

fbshipit-source-id: 3e8664ce2bc5159e464d1d76ed37294c4eac1709
2020-10-16 19:13:19 -07:00
Jun Wu
6dac514fae continue: pass --noninteractive to continued command
Summary: This preserves the `--noninteractive` flag used by some tools.

Reviewed By: DurhamG

Differential Revision: D24040789

fbshipit-source-id: 8d50f3f3ce6b2015f0ef6c3bd1b4fbb874d0ea7d
2020-10-16 18:40:51 -07:00
Jun Wu
b36584e704 configparser: set ui.merge:interactive from ui.merge in user config
Summary:
This restores the compatibility of setting up merge tools using the `ui.merge`
config while still limiting the default `editmerge` tool to interactive
sessions.

Reviewed By: sfilipco

Differential Revision: D24377259

fbshipit-source-id: 3d2befba412b824fc985ddffa131e339644178c2
2020-10-16 18:33:06 -07:00
Jun Wu
122108e46e configparser: move load_user to a testable method
Summary: Make it testable by allowing specifying paths to load as user hgrc.

Reviewed By: sfilipco

Differential Revision: D24377258

fbshipit-source-id: 969028df64d55ad1f1304e35675d84595ed6a2bf
2020-10-16 18:33:06 -07:00
Lukas Piatkowski
1b5293d584 mononoke/integration: create a Makefile to run tests as part of getdeps.py build (#67)
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/67

With this change it will be possible to build dependencies of and run integration tests using getdeps.py.

This is the first goal of Q4 as per https://fb.quip.com/v8YzAYNSYgot: "Get Open Source version of integration tests running on Legocastle".

Before this diff:
The OSS integration tests run now on GitHub by:
- Building some test dependencies with getdeps.py
- Building some test dependencies with homebrew/apt-get
- Running tests via python script

The OSS integration tests were not running on Sandcastle.

After this diff:
The OSS integration tests run on Github by:
- Building and executing tests via getdeps.py (execution of tests happens by getdeps.py calling Make calling python script)

The OSS integration tests run on Sandcastle using the same getdeps.py setup as Github.

Reviewed By: krallin

Differential Revision: D24253268

fbshipit-source-id: cae249b72d076222673b8bbe4ec21866dcdbb253
2020-10-16 12:23:33 -07:00