Summary:
Before replacing the existing debugInodeStatus changes, make some
small refactorings to help me understand it.
Reviewed By: xavierd
Differential Revision: D24299970
fbshipit-source-id: 3b60aa8365bb9a45e12357374282ba837ccc36eb
Summary:
Add a function that makes it easy to traverse EdenFS's view of the
traversed-so-far filesystem.
Reviewed By: xavierd
Differential Revision: D24299962
fbshipit-source-id: 275c849846bf45a2e80780411d60266b961d825b
Summary:
Adding number of changed files to mononoke_commit and mononoke_commit_draft scribe categories.
I need to create scuba table for mononoke_commit scribe category.
Reviewed By: krallin
Differential Revision: D24390738
fbshipit-source-id: 80b9917751a42377b49190cbb2f650a21b086e74
Summary:
- convert ManifestOps to new style futures
- at this point `//eden/manifest:manifest` crate is completely free from old style futures
Reviewed By: krallin
Differential Revision: D24502214
fbshipit-source-id: f1cdb11bd8234f22af5c905243f71e1e9fca11f1
Summary: `derive_manifest_*` functions are now based on new futures and use new bounded_traversal
Reviewed By: krallin
Differential Revision: D24448824
fbshipit-source-id: 5af06851f9c48a73e53daafe1f4972c186a5ca80
Summary: Somewhat more convenient to work with, no need to look inside.
Reviewed By: StanislavGlebik
Differential Revision: D24474898
fbshipit-source-id: 7ee0920e7d0d5a2102c68695a5cc0d9e237d958d
Summary:
Users get confused and try to pull from the wrong repo - e.g. `hg checkout D23665606` in `fbsource` - and find the resulting output about missing hashes scary.
Give them a nice clear error instead.
Reviewed By: wez
Differential Revision: D24485097
fbshipit-source-id: e9aeb7ed6aaf494f828d736ca5203100807fb321
Summary:
indexedlogdatastore is supposed to use remotefilelog.cachelimit to set
the max size, but instead it was setting the max per-log size, which means the
max size was N times bigger. Let's fix that.
Reviewed By: xavierd
Differential Revision: D24483181
fbshipit-source-id: f33cedbfdbb318e9d5eb9fda497645050b93e9fe
Summary:
This updates Thrift codegen for Rust to call hooks on a context stack in a
similar fashion as the C++ codegen.
There is still a bit of room for improvement here. Notably, we don't know how
much data we actually read or sent to the client, so for now those are
hard-coded to zero in the codegen. That's better than not calling those hooks
at all (which is what happens right now), but it could stand to be improved.
Reviewed By: jsgf
Differential Revision: D24445298
fbshipit-source-id: 470daf03057424dc300b6a193668be835ae28452
Summary:
In the next diff in this stack, I'm changing the Rust thrift library as well as
the codegen, and this is causing quite a bit of pain with regard to this
codegen that is checked in to the hg build:
- Regenerating the codegen isn't super obvious.
- The Sandcastle build fails because it uses the current codegen with the old
library.
According to lukaspiatkowski, this has also been a problem in the past when Eden got
migrated to Tokio 0.2, so I'd like to save myself and others some pain by just
not generating the server side codegen and not checking it in, since it's
unused. This reduces the surface of stuff that might go out of sync.
#forcetdhashing
Reviewed By: markbt
Differential Revision: D23649604
fbshipit-source-id: d684bec427431a366de42c88e53072caa98d5b2f
Summary:
Now that Linux/macOS startup no longer uses fork it becomes trivial to share
the same code with Windows. This improves Windows startup in several different ways:
- `edenfsctl start` now displays the status of the start process in the console
- `edenfsctl start` no longer has an arbitrary timeout after which it reports
a timeout even though EdenFS is still starting up.
This also kills a bunch of Windows specific code that is no longer needed.
Reviewed By: fanzeyi
Differential Revision: D24393690
fbshipit-source-id: 28100aec96da81c92d5b592353edceed332e2364
Summary:
Previously, when that code was ported on Windows, paths separator were
converted from '\' to '/' when a wide string was provided, all the other paths
were treated as is.
The main issue with this strategy is that not all paths can be converted, the
non-stored ones for instance are immutable, which leads to some subtle bugs
down the line. For instance, the paths: "Z:/foo/bar/baz" and "Z:\foo/bar\baz"
would not be equal as the path separator isn't the same, but both of these are
actually the same path underneath.
To solve this, this diff first introduce a Windows path separator, and then
modifies the path comparison functions to ignore the path separator and only
compare the components.
I'm definitively not a fan of the pattern I use for searching for both / and \
in paths, suggestions are welcome for how to improve that.
Reviewed By: chadaustin
Differential Revision: D24376980
fbshipit-source-id: 0702bf775c7c3937b2138abd5a63d339ac80aaed
Summary:
The path iterator functions are skipping sanity checking of paths as they
assume that the path has already been validated. Unfortunately, that isn't the
case as the only sanity checking we are doing is on the beginning and end of
the string that is passed in. Notably, `RelativePathPiece{"./foo"}` would not
fail the sanity checking even though it it's not made of valid components.
Reviewed By: chadaustin
Differential Revision: D24439804
fbshipit-source-id: 852b3a1180b185cb0bfb96bf5bcdc98b231f32c5
Summary:
As the string is created with MAX_PATH wide characters, a lot of them are going
to stay nul. A future diff adds a sanity check to the PathComponent that
validates that it doesn't contain any nul bytes, which chokes on these paths.
Reviewed By: chadaustin
Differential Revision: D24479715
fbshipit-source-id: e51c4b1c53a3f375664c0c0a1e325ebe302b0cd3
Summary:
When the new rust store is enabled, it can throw exceptions if the rust
code tries to fetch multiple trees when the server doesn't support bfs fetching.
The python code path ignores these requests, letting the eventual single tree
lookup path (i.e. not the prefetch path) perform the download. Let's let the
rust code do the same for rust stores.
Eventually every repo will support bfs fetching and we can delete this code path
entirely.
Reviewed By: quark-zju
Differential Revision: D24459106
fbshipit-source-id: d9a931364bd3235113b0dbcf09cba35a16e16e8e
Summary: Comment out the tests that do not compile on Windows.
Reviewed By: fanzeyi
Differential Revision: D24458438
fbshipit-source-id: 14bd4803ccd68ea6e02f95f868b8a801161518ea
Summary: let's make sure we don't create too big commits by setting the limit
Reviewed By: ikostia
Differential Revision: D24451569
fbshipit-source-id: 7b4f653509fce47d921e55dddad2de49ac5672a3
Summary:
This is a command that creates a commit that copies one directory into another.
Later you can take a commit and push it.
At the moment the command is very simple and also not very useful. At the very
least we need a way to limit the size of the commit, and this option is coming
in the next diff.
Note that the command doesn't overwrite the files if they exist in the target
directory - we may add an option to do so in future. Also if a file exists in
target directory but doesn't exist in source directory then it's left
untouched.
Reviewed By: ikostia
Differential Revision: D24451568
fbshipit-source-id: 2c7b945aadc99a1ca960349992c3684ef02ba35e
Summary: This merely filters out the files that don't compile on mode/win.
Reviewed By: chadaustin
Differential Revision: D24456917
fbshipit-source-id: d314ca2936e064133f2fc6a6199b38d17f2d9ef7
Summary:
FUSE_ACCESS was incorrectly categorized as an 'other' FUSE request
type. Categorize it as 'read'.
Reviewed By: wez
Differential Revision: D24437458
fbshipit-source-id: 5ea0a7d5a0fd57b9230f37674fd2568a4917bab5
Summary:
I noticed while playing with `eden strace` that, on my Linux machine,
running `ls -lA` would cause a pile of FUSE_GETXATTR requests for
`system.posix_acl_access` and `system.posix_acl_default`.
FUSE does not cache getxattr yet, but the kernel can cache ACLs if we
pretend to support them. Thus, advertise FUSE_POSIX_ACL support, but
continue returning ENOSYS for all setxattr calls.
Reviewed By: wez
Differential Revision: D24039130
fbshipit-source-id: a2e69c43b728fb51fb1d1b41ca9d31eba8efaa19
Summary:
Add an `eden strace` command that subscribes to all FUSE events for
the specified mount (or current directory), and streams them to stdout
with response latency measurements and the requesting process.
If there are any pending requests at the time `eden strace` is run,
they are printed first.
Reviewed By: wez
Differential Revision: D24038978
fbshipit-source-id: 59a2112dbdb1708571d12b04bdccaf9eca725cf7
Summary:
This is to be able to run the hook tailer on fbsource commits, which had been
push-redirected from ovrsource.
Reviewed By: StanislavGlebik
Differential Revision: D24447814
fbshipit-source-id: bfb164ec45dd2d4d5c3c83f133b48000a71a50a3
Summary: As well as the unixname of the pusher, log all identities provided by the pusher.
Reviewed By: krallin
Differential Revision: D24446453
fbshipit-source-id: 0854b3a10221da16d87b53fd849f033f7bb2a27e
Summary:
Use the new API `smartset.prefetch` for the prefetching logic for `log`
commands.
Prefetching is only enabled if the template contains related symbols.
Reviewed By: DurhamG
Differential Revision: D24289052
fbshipit-source-id: 810da3b249d45a7516aaf39a0b74f6965b87f96f
Summary: This makes `log` command more efficient using the `hybrid` backend.
Reviewed By: DurhamG
Differential Revision: D24293498
fbshipit-source-id: db3bac28a0b04ba6af449357bc08b285cd62780d
Summary:
Add prefetch APIs on smartset to make it easier to specify what need to be
prefetched.
This is intended to be used by `{phabstatus}`, log with lazy commit text backend,
and revsets like `author(x)`, etc.
Reviewed By: sfilipco
Differential Revision: D24321292
fbshipit-source-id: 7e645ccbe39feeae1e2da7bfb115623945bf9cef
Summary:
This makes it possible to "fork" sets and set different prefetch properties on
them.
Reviewed By: sfilipco
Differential Revision: D24365329
fbshipit-source-id: c4bdd6ab21e6a891a02d450f9e6edab3a3781eca
Summary:
The generatorset has a pure Python implementation for rewindable generator.
However it is not thread-safe, and we want thread-safety since the set
iteration will be driven by async Rust in multiple threads.
One of the issues is, during thread 1 `next(gen)`, thread 2 might call
`next(gen)`, and that's not allowed by Python.
Fix them by switching to the Rust RGenerator.
Reviewed By: DurhamG
Differential Revision: D24365328
fbshipit-source-id: 2785e80c7c460a7f754ed23e3af99f4a5c9fbcdf
Summary:
Buck-out can contain some files that we wont have the ability to read on mac os
(like .Spotlight-V100 and .Trashes). This means `eden du` will error out on
reading them.
We handle FileNotFound but not permission errors, lets treat them the same.
Reviewed By: genevievehelsel
Differential Revision: D24396252
fbshipit-source-id: 31e2ecfeff1783ae9c3479f59335d95a0d9b6de7
Summary:
Similarly to the indexedlogdatastore, the LFS stores can become corrupted on
power loss. This is happening fairly frequently on Windows Sandcastle due to
the OS being virtualized and power being cut abruptly.
For now this only attempts to repair the shared stores, in theory we could also
try to repair the local stores but haven't looked into it.
Reviewed By: DurhamG
Differential Revision: D24449202
fbshipit-source-id: 605a7943a0850b625bf00c514879b3da1ab2b406
Summary:
The ContentStore/Metadatastore are made of several different stores, attempting
to expose all of them to Python to drive the repair logic from there would leak
implementation detail of how the stores are implemented.
Instead, let's simply expose a single `repair` function out of the
pyrevisionstore crate that takes care of repairing all of the underlying
stores. For now, this is just moving code around, but a future diff will
integrate the LFS stores.
Reviewed By: DurhamG
Differential Revision: D24449203
fbshipit-source-id: 1631ced9068716453cb404bf7e65cefbf2db5247
Summary: My previous diff shrink the size of the cache so the pools can't fit there anymore. Adjust them as well. Plus made cache argument pareser understand float values for `cache-size-gb`
Reviewed By: johansglock
Differential Revision: D24448419
fbshipit-source-id: 2b73f789df10c5df7685ba96b7f19b8c9d04cc71
Summary: The hostname is of limited use when running on Tupperware - add in the session ID, which lets us look at what happened.
Reviewed By: StanislavGlebik
Differential Revision: D24361320
fbshipit-source-id: 0ac2c64d68915288b3241320c7ee14452b754ec8
Summary:
I removed --debug flag for bookmarks validator on tw because it was too spammy,
but now we get no output at all. Let's make it info so that we can see it in tw
output
Reviewed By: ikostia
Differential Revision: D24445829
fbshipit-source-id: d37c41be46794867f91ecf66f0318e7b2d660d85
Summary:
This will be used to avoid 1-by-1 fetching for the changelog backend with
commit text stored remotely.
Reviewed By: sfilipco
Differential Revision: D24321293
fbshipit-source-id: 9695c72166cadc0b167e2ce7fde822cdf6b1cea8
Summary:
The communication protocol between edenfs and watchman is that edenfs
advertises that "something in the mount has changed!" and it's up to
watchman to call getFilesChangedSince or getCurrentJournalPosition to
inspect what has changed.
This diff avoids redundantly notifying "something in the mount has
changed!" between the initial notification and actually reading the
journal contents. This way, a Watchman subscription cannot (*) fall
behind, and memory usage stays approximately constant during heavy
write traffic.
Hopefully, this will prevent the kernel from thinking the edenfs mount
is a slow IO device under high memory pressure.
* Technically, if there were a multitude of subscribeStreamTemporary
streams, and all but one was not calling getFilesChangedSince, it's
possible that notifications could back up on the stuck
subscriptions.
Reviewed By: wez
Differential Revision: D24090247
fbshipit-source-id: 6561c13e847b749c093adab75250df474d3210f9
Summary: This will be used to test if a keyword is in an AST.
Reviewed By: DurhamG
Differential Revision: D24293499
fbshipit-source-id: c4c238e09fae6728d9d20ffe94e04194ac20bbb8
Summary:
There were 2 recent reports about "visibleheads" not being enabled.
Previously, to enable visibility, one has to run a `pull` command with
`visibility.automirate=start` and `visibility.enabled=true`, and
`pull.automirate=true`, and our current config does not have
`visibility.automigrate`.
There are other issues with pull-based auto migration. For example, the edenfs
backing repo does not run pull and won't be migrated.
Move the visibility and dirstate automigrate logic to `repo.__init__` so they
always run. Also simplify the configuration so a single config
`visibility.enabled=true` (or `format.dirstate`) is enough to trigger the
migration.
Reviewed By: DurhamG
Differential Revision: D24340586
fbshipit-source-id: 2b8ed8da51c2345c9e6e680519dbc4f10eab1ac0
Summary:
We expect `.hg/store` to exist (ex. `storerequirements`). Drop support for repo
layout without a store.
Reviewed By: DurhamG
Differential Revision: D24340585
fbshipit-source-id: 757dc8633db9549a6bea082281326ebdefcbfcba
Summary:
Treestate is the only supported dirstate format. Remove test cases about legacy
dirstates.
Reviewed By: DurhamG
Differential Revision: D24340584
fbshipit-source-id: 2ebaa7d5aa9e8639e3ce4caa67c4f93155b52558
Summary:
Turn on rust changelog (changelog2) for all hosts (except hgsql).
Turn on doublewrite backend for hg-dev hosts, triggered by pull.
Tests are mostly working, and I have been using it for weeks.
Reviewed By: singhsrb
Differential Revision: D24259759
fbshipit-source-id: b89a27f98a6d3d1e4ea187bf7b29f875d0e96e2e
Summary:
Include opcode name and process name in eden.FuseCall Thrift structs
so we can use eden.FuseCall in a later diff in the stack.
Reviewed By: kmancini
Differential Revision: D24036420
fbshipit-source-id: fc6d8f3d174b85e07fac299a6f86b2b2d24f301d
Summary: It will be used to test whether `{phabstatus}` is in template.
Reviewed By: sfilipco
Differential Revision: D24293496
fbshipit-source-id: 1b59236e07280c0cefb07be9da7a3fd18d9940a1
