Commit Graph

61327 Commits

Author SHA1 Message Date
Chad Austin
7ec5741e06 indicate access() is a read FUSE call
Summary:
FUSE_ACCESS was incorrectly categorized as an 'other' FUSE request
type. Categorize it as 'read'.

Reviewed By: wez

Differential Revision: D24437458

fbshipit-source-id: 5ea0a7d5a0fd57b9230f37674fd2568a4917bab5
2020-10-22 11:57:11 -07:00
Chad Austin
37f030ba72 optimize ACL lookups by pretending to support FUSE_POSIX_ACL
Summary:
I noticed while playing with `eden strace` that, on my Linux machine,
running `ls -lA` would cause a pile of FUSE_GETXATTR requests for
`system.posix_acl_access` and `system.posix_acl_default`.

FUSE does not cache getxattr yet, but the kernel can cache ACLs if we
pretend to support them. Thus, advertise FUSE_POSIX_ACL support, but
continue returning ENOSYS for all setxattr calls.

Reviewed By: wez

Differential Revision: D24039130

fbshipit-source-id: a2e69c43b728fb51fb1d1b41ca9d31eba8efaa19
2020-10-22 11:57:11 -07:00
Chad Austin
c2e9a4acd7 eden strace
Summary:
Add an `eden strace` command that subscribes to all FUSE events for
the specified mount (or current directory), and streams them to stdout
with response latency measurements and the requesting process.

If there are any pending requests at the time `eden strace` is run,
they are printed first.

Reviewed By: wez

Differential Revision: D24038978

fbshipit-source-id: 59a2112dbdb1708571d12b04bdccaf9eca725cf7
2020-10-22 11:39:49 -07:00
svcscm
0e368d7b37 Updating submodules
Summary:
GitHub commits:

eaee7b4bae
e6f158c56d
f8aa1bd98c
6a782bda39

Reviewed By: yns88

fbshipit-source-id: 08dd6c610a21e939a43eee3f2a56af839997154e
2020-10-22 11:09:24 -07:00
Kostia Balytskyi
1710662e9e hook_tailer: be able to run on push-redirected commits
Summary:
This is to be able to run the hook tailer on fbsource commits, which had been
push-redirected from ovrsource.

Reviewed By: StanislavGlebik

Differential Revision: D24447814

fbshipit-source-id: bfb164ec45dd2d4d5c3c83f133b48000a71a50a3
2020-10-22 02:48:18 -07:00
Kostia Balytskyi
65f806fa2a cross_repo_sync: don't expect Option<CommitSyncConfigVersion>
Summary: We know it's there.

Reviewed By: StanislavGlebik

Differential Revision: D24449015

fbshipit-source-id: 3ce402b21ce1b8bd4f28980d42f86651bf77c68f
2020-10-22 02:44:09 -07:00
Mark Thomas
253d8253b9 scribe_commit_queue: log all identities of pushing user
Summary: As well as the unixname of the pusher, log all identities provided by the pusher.

Reviewed By: krallin

Differential Revision: D24446453

fbshipit-source-id: 0854b3a10221da16d87b53fd849f033f7bb2a27e
2020-10-22 01:11:11 -07:00
Mark Thomas
f31d6918a5 scribe_commit_queue: fix deps lint
Reviewed By: StanislavGlebik

Differential Revision: D24447560

fbshipit-source-id: 426af3393b6418ad29f43ef2152f3995aace0cc1
2020-10-22 01:11:10 -07:00
svcscm
bc8b8a4241 Updating submodules
Summary:
GitHub commits:

db71ef8cf9

Reviewed By: yns88

fbshipit-source-id: 12241b7ef903173015fa8d9ea6ae896066ea1e3e
2020-10-22 01:11:10 -07:00
svcscm
6fcdba8845 Updating submodules
Summary:
GitHub commits:

298b697f65

Reviewed By: yns88

fbshipit-source-id: 11825dabd8b6b4edcbaceab91bd7cd18b2061302
2020-10-21 18:43:46 -07:00
Jun Wu
a5136d1f18 phabstatus: prefetch diff status using new smartset.prefetch APIs
Summary:
Use the new API `smartset.prefetch` for the prefetching logic for `log`
commands.

Prefetching is only enabled if the template contains related symbols.

Reviewed By: DurhamG

Differential Revision: D24289052

fbshipit-source-id: 810da3b249d45a7516aaf39a0b74f6965b87f96f
2020-10-21 17:49:15 -07:00
Jun Wu
ea03d62698 log: prefetch commit data by template
Summary: This makes `log` command more efficient using the `hybrid` backend.

Reviewed By: DurhamG

Differential Revision: D24293498

fbshipit-source-id: db3bac28a0b04ba6af449357bc08b285cd62780d
2020-10-21 17:49:15 -07:00
Jun Wu
afa9da8ff8 smartset: add prefetch APIs
Summary:
Add prefetch APIs on smartset to make it easier to specify what need to be
prefetched.

This is intended to be used by `{phabstatus}`, log with lazy commit text backend,
and revsets like `author(x)`, etc.

Reviewed By: sfilipco

Differential Revision: D24321292

fbshipit-source-id: 7e645ccbe39feeae1e2da7bfb115623945bf9cef
2020-10-21 17:49:15 -07:00
Jun Wu
9f0d3e4745 smartset: add clone API
Summary:
This makes it possible to "fork" sets and set different prefetch properties on
them.

Reviewed By: sfilipco

Differential Revision: D24365329

fbshipit-source-id: c4bdd6ab21e6a891a02d450f9e6edab3a3781eca
2020-10-21 17:49:15 -07:00
Jun Wu
9f664a8b30 smartset: use Rust reentrant generator for generatorset
Summary:
The generatorset has a pure Python implementation for rewindable generator.
However it is not thread-safe, and we want thread-safety since the set
iteration will be driven by async Rust in multiple threads.

One of the issues is, during thread 1 `next(gen)`, thread 2 might call
`next(gen)`, and that's not allowed by Python.

Fix them by switching to the Rust RGenerator.

Reviewed By: DurhamG

Differential Revision: D24365328

fbshipit-source-id: 2785e80c7c460a7f754ed23e3af99f4a5c9fbcdf
2020-10-21 17:17:08 -07:00
Katie Mancini
2721a17907 Handle permission errors in eden du
Summary:
Buck-out can contain some files that we wont have the ability to read on mac os
(like .Spotlight-V100 and .Trashes). This means `eden du` will error out on
reading them.

We handle FileNotFound but not permission errors, lets treat them the same.

Reviewed By: genevievehelsel

Differential Revision: D24396252

fbshipit-source-id: 31e2ecfeff1783ae9c3479f59335d95a0d9b6de7
2020-10-21 17:00:10 -07:00
svcscm
ee6f948ae2 Updating submodules
Summary:
GitHub commits:

9ce17be977
f817aff73b
5227b315ec

Reviewed By: yns88

fbshipit-source-id: e59f909bb90629b64147110b427cf00a36d08544
2020-10-21 17:00:10 -07:00
svcscm
2afd2e32fa Updating submodules
Summary:
GitHub commits:

a55fdc39aa

Reviewed By: yns88

fbshipit-source-id: 44b1e15a32dbe23b339fc168c0c48e3e2820cead
2020-10-21 14:17:41 -07:00
Xavier Deguillard
cd549f96bb revisionstore: repair the LFS store during hg doctor
Summary:
Similarly to the indexedlogdatastore, the LFS stores can become corrupted on
power loss. This is happening fairly frequently on Windows Sandcastle due to
the OS being virtualized and power being cut abruptly.

For now this only attempts to repair the shared stores, in theory we could also
try to repair the local stores but haven't looked into it.

Reviewed By: DurhamG

Differential Revision: D24449202

fbshipit-source-id: 605a7943a0850b625bf00c514879b3da1ab2b406
2020-10-21 13:20:51 -07:00
Xavier Deguillard
c62cd9e8c8 revisionstore: move the repair logic from doctor
Summary:
The ContentStore/Metadatastore are made of several different stores, attempting
to expose all of them to Python to drive the repair logic from there would leak
implementation detail of how the stores are implemented.

Instead, let's simply expose a single `repair` function out of the
pyrevisionstore crate that takes care of repairing all of the underlying
stores. For now, this is just moving code around, but a future diff will
integrate the LFS stores.

Reviewed By: DurhamG

Differential Revision: D24449203

fbshipit-source-id: 1631ced9068716453cb404bf7e65cefbf2db5247
2020-10-21 13:20:51 -07:00
svcscm
78274d6d18 Updating submodules
Summary:
GitHub commits:

1f1c1193e1
9b9237a40b

Reviewed By: yns88

fbshipit-source-id: 199e9cc3f9c64841817764e2df5caea806dcf436
2020-10-21 12:52:32 -07:00
Egor Tkachenko
2494f1db7e Fix cache pool sizes
Summary: My previous diff shrink the size of the cache so the pools can't fit there anymore. Adjust them as well. Plus made cache argument pareser understand float values for `cache-size-gb`

Reviewed By: johansglock

Differential Revision: D24448419

fbshipit-source-id: 2b73f789df10c5df7685ba96b7f19b8c9d04cc71
2020-10-21 10:16:45 -07:00
Simon Farnsworth
4df44dae29 Add the session ID into the clienttelemetry hostname
Summary: The hostname is of limited use when running on Tupperware - add in the session ID, which lets us look at what happened.

Reviewed By: StanislavGlebik

Differential Revision: D24361320

fbshipit-source-id: 0ac2c64d68915288b3241320c7ee14452b754ec8
2020-10-21 08:59:47 -07:00
Stanislau Hlebik
d4326273a7 mononoke: change debug logging to info
Summary:
I removed --debug flag for bookmarks validator on tw because it was too spammy,
but now we get no output at all. Let's make it info so that we can see it in tw
output

Reviewed By: ikostia

Differential Revision: D24445829

fbshipit-source-id: d37c41be46794867f91ecf66f0318e7b2d660d85
2020-10-21 05:53:09 -07:00
svcscm
e629afcbe8 Updating submodules
Summary:
GitHub commits:

d6ed6a77e2
c0a75500df
c17e84049e
afced0bf7b
e5af792870

Reviewed By: yns88

fbshipit-source-id: 0820e4ffd85614f1166e8479dd0598edbf9aad59
2020-10-21 03:41:17 -07:00
svcscm
2d7fc0dc75 Updating submodules
Summary:
GitHub commits:

fe3b43ea49
8345649615
739171fb18

Reviewed By: yns88

fbshipit-source-id: a735f937a6eb7c06a94d0a493d9d8142876a9bbb
2020-10-21 03:41:17 -07:00
svcscm
715854d105 Updating submodules
Summary:
GitHub commits:

5649047eff

Reviewed By: yns88

fbshipit-source-id: 93e56f484669653dea4589077da4cbfc88f7505c
2020-10-20 19:36:51 -07:00
Jun Wu
91ab519edb streams: add API to prefetch commit text in a streaming fashion
Summary:
This will be used to avoid 1-by-1 fetching for the changelog backend with
commit text stored remotely.

Reviewed By: sfilipco

Differential Revision: D24321293

fbshipit-source-id: 9695c72166cadc0b167e2ce7fde822cdf6b1cea8
2020-10-20 18:40:58 -07:00
Chad Austin
76409f59b7 avoid redundant change notifications when prior notifications have not been observed
Summary:
The communication protocol between edenfs and watchman is that edenfs
advertises that "something in the mount has changed!" and it's up to
watchman to call getFilesChangedSince or getCurrentJournalPosition to
inspect what has changed.

This diff avoids redundantly notifying "something in the mount has
changed!" between the initial notification and actually reading the
journal contents. This way, a Watchman subscription cannot (*) fall
behind, and memory usage stays approximately constant during heavy
write traffic.

Hopefully, this will prevent the kernel from thinking the edenfs mount
is a slow IO device under high memory pressure.

* Technically, if there were a multitude of subscribeStreamTemporary
  streams, and all but one was not calling getFilesChangedSince, it's
  possible that notifications could back up on the stuck
  subscriptions.

Reviewed By: wez

Differential Revision: D24090247

fbshipit-source-id: 6561c13e847b749c093adab75250df474d3210f9
2020-10-20 18:25:43 -07:00
Jun Wu
d86802a980 parser: add a function to walk through a tree
Summary: This will be used to test if a keyword is in an AST.

Reviewed By: DurhamG

Differential Revision: D24293499

fbshipit-source-id: c4c238e09fae6728d9d20ffe94e04194ac20bbb8
2020-10-20 16:45:43 -07:00
Jun Wu
5a6161f140 repo: unconditionally migrate to specified visibility and dirstate format
Summary:
There were 2 recent reports about "visibleheads" not being enabled.

Previously, to enable visibility, one has to run a `pull` command with
`visibility.automirate=start` and `visibility.enabled=true`, and
`pull.automirate=true`, and our current config does not have
`visibility.automigrate`.

There are other issues with pull-based auto migration. For example, the edenfs
backing repo does not run pull and won't be migrated.

Move the visibility and dirstate automigrate logic to `repo.__init__` so they
always run. Also simplify the configuration so a single config
`visibility.enabled=true` (or `format.dirstate`) is enough to trigger the
migration.

Reviewed By: DurhamG

Differential Revision: D24340586

fbshipit-source-id: 2b8ed8da51c2345c9e6e680519dbc4f10eab1ac0
2020-10-20 16:36:50 -07:00
Jun Wu
9de7c7a22d repo: drop format.usestore=false support
Summary:
We expect `.hg/store` to exist (ex. `storerequirements`). Drop support for repo
layout without a store.

Reviewed By: DurhamG

Differential Revision: D24340585

fbshipit-source-id: 757dc8633db9549a6bea082281326ebdefcbfcba
2020-10-20 16:36:50 -07:00
Jun Wu
e179789b7b tests: remove test cases about legacy dirstates
Summary:
Treestate is the only supported dirstate format. Remove test cases about legacy
dirstates.

Reviewed By: DurhamG

Differential Revision: D24340584

fbshipit-source-id: 2ebaa7d5aa9e8639e3ce4caa67c4f93155b52558
2020-10-20 16:36:50 -07:00
svcscm
36e5276ba9 Updating submodules
Summary:
GitHub commits:

e53bda6f10
55fdba7f48
6d56669bb1

Reviewed By: yns88

fbshipit-source-id: 257d47e6b40526dca2285f751fe53e5176590a73
2020-10-20 16:05:11 -07:00
Jun Wu
a3ac43a23a rollout: use doublewrite changelog backend for hg-dev
Summary:
Turn on rust changelog (changelog2) for all hosts (except hgsql).

Turn on doublewrite backend for hg-dev hosts, triggered by pull.
Tests are mostly working, and I have been using it for weeks.

Reviewed By: singhsrb

Differential Revision: D24259759

fbshipit-source-id: b89a27f98a6d3d1e4ea187bf7b29f875d0e96e2e
2020-10-20 15:53:57 -07:00
Chad Austin
d5c04ab941 report opcode and process names for outstanding fuse calls
Summary:
Include opcode name and process name in eden.FuseCall Thrift structs
so we can use eden.FuseCall in a later diff in the stack.

Reviewed By: kmancini

Differential Revision: D24036420

fbshipit-source-id: fc6d8f3d174b85e07fac299a6f86b2b2d24f301d
2020-10-20 15:35:47 -07:00
Jun Wu
7db31b57b6 templater: add API to parse string to AST with aliases expanded
Summary: It will be used to test whether `{phabstatus}` is in template.

Reviewed By: sfilipco

Differential Revision: D24293496

fbshipit-source-id: 1b59236e07280c0cefb07be9da7a3fd18d9940a1
2020-10-20 15:24:29 -07:00
Jun Wu
c9d716fd51 test-rebase-emptycommit: enable segmented changelog
Reviewed By: singhsrb

Differential Revision: D24270422

fbshipit-source-id: d2dcf6db2a0c8ebd90a2edb3ee1149d8f9bd4fbe
2020-10-20 15:24:29 -07:00
Jun Wu
063bc373af test-rebase-check-restore: enable segmented changelog
Reviewed By: singhsrb

Differential Revision: D24270424

fbshipit-source-id: 2833dbbdc7ec34375f070fc6ce273265272a205e
2020-10-20 15:24:29 -07:00
Jun Wu
293d05dd40 test-rebase-conflicts: enable segmented changelog
Reviewed By: singhsrb

Differential Revision: D24258215

fbshipit-source-id: 9a9bba13c3963c79df5e672e5ac011ab7c7e5698
2020-10-20 15:24:29 -07:00
Jun Wu
20ac277bc5 test-rebase-brute-force: enable segmented changelog
Reviewed By: singhsrb

Differential Revision: D24258213

fbshipit-source-id: aceb6e0792b4646eb0805f56d981751eb85ae340
2020-10-20 15:24:29 -07:00
Jun Wu
af38cf3cb9 test-rebase-abort: enable segmented changelog
Reviewed By: singhsrb

Differential Revision: D24258214

fbshipit-source-id: db7ecb6b874da81d0e553cd9f1514ee5559635b7
2020-10-20 15:24:29 -07:00
Jun Wu
d39be1d667 tests: mark more tests as compatible with segmented changelog
Reviewed By: singhsrb

Differential Revision: D24214459

fbshipit-source-id: 7c125cbc2c0d559d337bd01107b784002e2dc9ea
2020-10-20 15:24:29 -07:00
Jun Wu
b46a5b6192 copies: improve compatibility with segmented changelog
Summary:
There are 2 issues:
- len(repo) is no longer the "max rev" + 1. Use 1<<63 instead.
- If invalidatelinkrev is a repo requirement, linkrev can no longer be trusted.

Reviewed By: DurhamG

Differential Revision: D24270423

fbshipit-source-id: 4e5c999772fad6ef2b1dabd2d6363d4226242c4d
2020-10-20 15:24:28 -07:00
Jun Wu
62f30ea23e changegroup: do not use bad linkrev to filter out file revisions
Summary:
If linkrevs are known unreliable, do not use them. This fixes some cases with
segmented changelog, file revisions are incorrectly filtered out and not sent
to client.

Reviewed By: DurhamG

Differential Revision: D24233228

fbshipit-source-id: 268a95b44c0217658b849317cb0a680450b91d23
2020-10-20 15:24:28 -07:00
Jun Wu
a7ba754d88 rebase: remove more uses of revnums
Summary: Replace revnum with commit hashes.

Reviewed By: singhsrb

Differential Revision: D24233229

fbshipit-source-id: 98b4d45b542b65c375cef405b45b66cd2dad22e4
2020-10-20 15:24:28 -07:00
Jun Wu
a4b3f573ba changelog: change "ambiguous prefix" to RepoLookupError
Summary: `LookupError` is too broad.

Reviewed By: DurhamG

Differential Revision: D24411203

fbshipit-source-id: 3e7c51618115c11f87f0fed670d18b4b5a03e20d
2020-10-20 15:24:28 -07:00
Jun Wu
8fe79db4b1 context: do not warn broken dirstate repetitively
Summary: This makes the output less verbose.

Reviewed By: DurhamG

Differential Revision: D24411259

fbshipit-source-id: 50b175f2e31ffda62e78dd226775edaf53893bce
2020-10-20 15:24:28 -07:00
Jun Wu
79764c4e04 changelog2: track "tip" explicitly
Summary:
With segmented changelog, the maximum revision is not always the "tip".
Therefore, track "tip" explicitly in svfs / metalog. This improves
compatibility.

Reviewed By: DurhamG

Differential Revision: D24214711

fbshipit-source-id: 7b22de7ac684a90965a695924bdb9114758aae43
2020-10-20 15:24:28 -07:00
Jun Wu
3a96a2a3ee hgcommits: stabilize order of inserting commits
Summary: Avoid HashSet or HashMap order to preserve the order of inserting commits.

Reviewed By: DurhamG

Differential Revision: D24214460

fbshipit-source-id: 66df2e0aba1820e6585f8da66897078f38abf82f
2020-10-20 15:24:28 -07:00