Summary:
This adds a debug command to blow away all RocksDB information that
can be reproduced from Mercurial. We will use it to help an Eden user
recover from a corrupted blob.
Reviewed By: bolinfest
Differential Revision: D8108649
fbshipit-source-id: 056dec19d51b9e430b3c2a249747b26830cfc875
Summary:
Add a clearCaches function to LocalStore that deletes all data from
LocalStore that could be retrieved from Mercurial.
Reviewed By: wez
Differential Revision: D8101365
fbshipit-source-id: d46d0db94e6f85aaf542d9f6b9b96fbdcc548b57
Summary:
The Overlay is the natural home for nextInodeNumber_ now that every
directory allocates inode numbers for its children right away. This
also simplifies serializing nextInodeNumber_ to disk in the following
diff.
Reviewed By: simpkins
Differential Revision: D8192442
fbshipit-source-id: 9b776a73c8d7653002b55985d592b1746e52f878
Summary:
There are a couple benefits to having an explicit Overlay close
method. First, it lets us assert if any code tries to access storage
after EdenMount has started shutting down. Also, it lets us pass
nextInodeNumber_ into Overlay::close() so it can be remembered in the
case of a clean shutdown.
Reviewed By: simpkins
Differential Revision: D8170534
fbshipit-source-id: 14fa607d877df8a3be9e9b0144132c8e8643d98d
Summary:
Before making changes to the way we record the maximum inode number,
cover it with tests.
Reviewed By: simpkins
Differential Revision: D7912059
fbshipit-source-id: 29fdc88bd74c6c07e870017d743d9fd0797baaf3
Summary:
Most filesystems limit path components to 255. To remain consistent,
let's have Eden do the same, at least for write operations.
Reviewed By: simpkins
Differential Revision: D8151020
fbshipit-source-id: 251da94a076f5765111c8e3d9d8a25c37682e2e3
Summary:
Add a new integration test that modifies a file, commits it, then restores it
to its original contents and creates a new commit with the revert.
This test isn't really exercising any Eden-specific behavior. The only reason
it is is interesting is that this happened to trigger a bug in the underlying
mercurial cdatapack code that caused the test to fail when run in treemanifest
mode. The mercurial cdatapack bug is fixed in D8131020
Reviewed By: quark-zju
Differential Revision: D7826284
fbshipit-source-id: 32da2db04452d5df0527f6be00b0c4c1a56c900b
Summary:
Update find_executables.py to be able to find the HG wrapper and binary using
buck rules. This is necessary in order to be able to run the integration
tests manually rather than via `buck test`. Without this the tests try to use
`/bin/hg.real` which tests the wrong version of the mercurial code and the
eden extension.
Reviewed By: bolinfest
Differential Revision: D8200168
fbshipit-source-id: d2a68c765c7143bbe613f32eef1268ed567790a7
Summary:
CodeMod: `future-then-continuation-lvalue-ref`.
```
buck run mode/opt foundation/clangr:clangr_coordinator -- run foundation/clang/codemods:future-then-continuation-lvalue-ref --dev -p 7 --num-workers=200
```
There is no need to support continuations taking non-const lvalue-ref params; callers can get exactly the same semantics with non-const rvalue-ref params instead. Thsi codemod preserves semantics. Many of these cases could do just as well taking by value, but this codemod preserves semantics.
Reviewed By: Orvid
Differential Revision: D8166008
fbshipit-source-id: 84b635cccce24b6485ffb5b24b98f10422b9a517
Summary: Mostly empty lines removed and added. A few bugfixes on excessive line splitting.
Reviewed By: cooperlees
Differential Revision: D8198776
fbshipit-source-id: 4361faf4a2b9347d57fb6e1342c494575f2beb67
Summary:
Update the hg_import_helper script to use the `repo._prefetchtrees()` method if
we need to fetch trees for a non-root tree. The "public"
`repo.prefetchtrees()` API does not allow specifying the path to the tree in
question.
Reviewed By: wez
Differential Revision: D8156733
fbshipit-source-id: 0b0b5e32a84b282d2d76e10c6bdef0cd807f5a73
Summary: We are changing `folly::collectAll` to return `SemiFuture` rather than `Future` and this is needed as an interim step. After all calls to `collectAll` are changed to `collectAllSemiFuture`, we'll be renaming it back to `collectAll`.
Reviewed By: yfeldblum
Differential Revision: D8157548
fbshipit-source-id: 27b768ac7ff0d6572bde57f01601045a1fd5d5e5
Summary:
This removes the main point of contention for eden prefetch
in two ways:
1. We batch up the complete list of blobs so that they can be processed
in bulk rather than stalling the tree walk
2. We can ask remotefilelog to check and fetch that list to the local
hgcache, again as a batch, rather than by forcing the data to be
loaded through into the local store
The goal of this prefetch is to bulk load data from the mercurial server
so that a subsequent file access doesn't have to make a one-off ssh session
for each one, rather than making sure that all the data is loaded into
the local store.
Reviewed By: chadaustin
Differential Revision: D7965818
fbshipit-source-id: 753400460d633b5467c5110e3f5608ce06106e00
Summary:
This is a first pass at a prefetcher. The idea is simple,
but the execution is impeded by some unfortunate slowness in different
parts of mercurial.
The idea is that you pass a list of glob patterns and we'll do something
to make accessing files that match those patterns ideally faster than
if you didn't give us the prefetch hint.
In theory we could run `hg prefetch -I PATTERN` for this, but prefetch
takes several minutes materializing and walking the whole manifest to
find matches, checking outgoing revs and various other overheads.
There is a revision flag that can be specified to try to reduce this
effort, but it still takes more than a minute.
This diff:
* Removes a `Future::get()` call in the GlobNode code
* Makes `globFiles` use Futures directly rather than `Future::get()`
* Adds a `prefetchFiles` parameter to `globFiles`
* Adds `eden prefetch` to the CLI and makes it call `globFiles` with
`prefetchFiles=true`
* Adds the abillity to glob over `Tree` as well as the existing `TreeInode`.
This means that we can avoid allocating inodes for portions of the
tree that have not yet been loaded.
When `prefetchFiles` is set we'll ask ObjectStore to load the blob for
matching files. I'm not currently doing this in the `TreeInode` case
on the assumption that we already did this earlier when its `TreeInode::prefetch`
method was called.
The glob executor joins the blob prefetches at each GlobNode level. It may
be possible to observe higher throughput if we join the complete set at the
end.
Reviewed By: chadaustin
Differential Revision: D7825423
fbshipit-source-id: d2ae03d0f62f00090537198095661475056e968d
Summary:
When calling mercurial from inside the Eden CLI, use `$EDEN_HG_BINARY` as the
path to hg if this variable is set. If it is not set, continue the current
behavior of using `hg` from the user's `$PATH`. This is primarily used during
the `eden clone` command.
This makes sure the Eden integration tests use `hg` built from the local
repository rather than the version of `hg` installed on the system. Most
locations in the integration tests were already doing so, but `eden clone` was
one place that still ended up using the system hg binary.
Reviewed By: wez
Differential Revision: D7839850
fbshipit-source-id: da801fad0767a111b3e3dfd393d82da8e2147e22
Summary:
Relax the restriction on changing uid/gid on inodes. We'll see what
cans of worms this opens I guess. (Landing this is low priority, but
might be important for making some of the existing tooling in fbsource
and www work.)
Reviewed By: simpkins
Differential Revision: D7768655
fbshipit-source-id: 95fe02fe7ddc001335dbdb34e16a989a85820240
Summary:
Add methods to UnixSocket and FutureUnixSocket to attach and detach from an
EventBase. This makes it possible to construct a UnixSocket object without
having an EventBase yet and then attach it to an EventBase later.
Reviewed By: bolinfest
Differential Revision: D8053423
fbshipit-source-id: c4de00166dbc0e075b4e4cd81c3dd5b377ea9a52
Summary:
Fix the code in FuseChannel::processSession() to look at the errno value
instead of the return value from `read()` when checking for ENODEV.
I accidentally broke this in D7436867. This wasn't causing any major issues,
since we would still break out of the loop, but it caused us to incorrectly
print warning messages about receiving an unexpected error. We would also
print this error message once per FUSE thread rather than just once for the
mount point.
Reviewed By: bolinfest
Differential Revision: D8109889
fbshipit-source-id: 9a53ca47b436ccf6731144ee2b829131339b6445
Summary: Notice this while checking something for a new hire yesterday.
Reviewed By: bolinfest
Differential Revision: D8119130
fbshipit-source-id: 8a27797061b2316c62f72e34c9f20130d97dc2b1
Summary:
This is used to dump the raw `JournalDelta` entries in the journal.
Hopefully this will help us figure out what's happening in T28686395, or more
generally, why we see get those merge errors that appear in the logs of the
form:
```
Journal for .hg/rebasestate holds invalid Created, Created sequence
```
from `eden/fs/journal/JournalDelta.cpp`.
(Note: this ignores all push blocking failures!)
Reviewed By: wez
Differential Revision: D7855071
fbshipit-source-id: f195813695bec7426329a9aacd84a9b1613feec9
Summary:
The calls to `getFilesChangedSince()` seem to fill up our logs without providing
a ton of value, so let's drop the default log level to `DBG3` so they do not
make it into the logs by default.
Reviewed By: chadaustin
Differential Revision: D8107917
fbshipit-source-id: 069673e9740487d8e4b62597beb13d5779971a79
Summary:
Add a `remove` command to the Eden CLI. This behaves like
`eden unmount --destroy`, but calling this "remove" is hopefully a more
intuitive UI. If stdin is a TTY this command also prompts the user for
confirmation before removing the checkout.
I plan to deprecate the `eden unmount --destroy` command in a subsequent
diff.
Reviewed By: wez
Differential Revision: D8086823
fbshipit-source-id: 562cf0f998eea416b80589b188eee255a10b9699
Summary: Now that permissions on directories work, verify umask works as intended.
Reviewed By: simpkins
Differential Revision: D7783743
fbshipit-source-id: 635221cd3255cc20e9ffa26b6838922c4a4110f3
Summary:
Unify how inode metadata is modified across inode types. This allows
changing the permission bits on a directory.
Reviewed By: simpkins
Differential Revision: D7767254
fbshipit-source-id: 35e9cf652c84c7d8680cc22dec7942e94e9f5af1
Summary:
This moves most inode metadata management into InodeBase and
persists permission bits (and eventually uid/gid) across Eden runs.
Reviewed By: simpkins
Differential Revision: D7035163
fbshipit-source-id: 50145449b56aad1662d53156e6e4960c5f7b6166
Summary: Store tree and file timestamps in the InodeTable so they persist across runs.
Reviewed By: simpkins
Differential Revision: D6891479
fbshipit-source-id: 1c9e6266375aceeaf293a81e73cf7f5334dbc32d
Summary:
This is stacked on top of Black 18.5b0.
allow-large-files
Reviewed By: carljm
Differential Revision: D8061834
fbshipit-source-id: 92e3645e159b60d77cf7e0bec64a8262ca4e88c2
Summary:
This is not at all clear from cppreference.com, but per
https://www.youtube.com/watch?v=dTeKf5Oek2c, it sounds to me like
recommended practice is to either:
`using namespace std::chrono_literals` (or string_literals or
whatever) to pull in a focused set of literals.
or
`using namespace std::literals` to pull in all standard literals.
or
`using namespace std` to pull in everything.
`using namespace std::literals::chrono_literals` is unnecessarily
verbose.
Adopt those standards in Eden.
Reviewed By: simpkins
Differential Revision: D8060944
fbshipit-source-id: 4d9dd4329698b7ff5e5c81b5b28780ca4d81a2a1
Summary:
I'd misunderstood the point of SharedMutex's upgrade locks -
unless they're used in rare paths, they don't allow for increased
concurrency. This diff and D7885245 remove all of Eden's ulocks,
replacing them with a helper which checks once with an rlock held, and
if the check fails, switches to a wlock (and checks again).
Reviewed By: yfeldblum
Differential Revision: D7886046
fbshipit-source-id: 545bb0dbb4898cbb71412efc6222ef12e4ee374e
Summary:
This allows multiple inodes to update their metadata simultaneously
without contending on InodeTable's locks.
Reviewed By: simpkins
Differential Revision: D7885245
fbshipit-source-id: cc8ab6cd90b7424beec314a115852e08b64026ae
Summary:
igorsugak reported an LSAN failure in Eden. Relatively straightforward to
track down - pending requests could sustain an Inode which ultimately
sustains EdenMount which owns the BackingStore, resulting in a
reference cycle for the duration of a fetch request. During EdenMount
shutdown in general we may want to consider aborting or discarding any
pending fetch requests.
Reviewed By: bolinfest, igorsugak
Differential Revision: D8047650
fbshipit-source-id: 32dafaff60570cf54a74ca4f57da61b6657d8ccb
Summary:
Some tools/script may be blindly assuming the presence of this file,
so rather than fight them, just create it.
Reviewed By: simpkins
Differential Revision: D8020274
fbshipit-source-id: 712e3bf31a0aefe27cc20f5361a0edb59c7deb9f
Summary:
Update the folly::Init code to define a `--logging` command line flag, and call
`folly::initLoggingOrDie()` with the value of this command line during
initialization.
This is similar to the existing code that initializes the glog library.
(Programs can use both glog and folly logging together in the same program, and
I expect that many programs will do so as parts get converted to folly::logging
and parts remain using glog.)
Reviewed By: yfeldblum
Differential Revision: D7827344
fbshipit-source-id: 8aa239fbad43bc0b551cbe40cad7b92fa97fcdde
Summary:
This updates the Eden mercurial extension to no longer invoke the Eden
`resetParentCommits()` thrift call when `setparents()` is called on the
dirstate map. Instead we now defer the call to `resetParentCommits()` until
`write()` is called to write the dirstate data to disk.
Informing edenfs of the parent change as soon as `setparents()` was called was
problematic, as this made edenfs reflect the change before the transaction was
committed. Some mercurial commands, notably `hg status` also call
`setparents()` on the dirstate but never write this back to disk at all. This
is problematic since `hg status` calls `setparents()` without holding any
mercurial locks. As a result it may call `setparents()` with the "wrong"
parent if another mercurial process is running and is in the middle of a
transaction.
Reviewed By: bolinfest, chadaustin
Differential Revision: D7980375
fbshipit-source-id: 4f5e4391fd291d4ea5fc93bb9d49ed0380fc1721
Summary:
While running the secfs filesystem validation tests against Eden, I
discovered a test that caused the eden process to abort. I bisected
and found that D7451330 regressed renaming a directory onto an empty
one. This fixes that case.
Reviewed By: simpkins
Differential Revision: D7945727
fbshipit-source-id: 592ede1b391528c02cd12b2b6ebbf3733fe8f503
Summary:
TestMount used to call through the EdenDispatcher when creating nodes
in the filesystem which had the unfortunate side effect of
incrementing the FUSE refcount. This diff asserts the FUSE refcounts
remain zero after inodes are created.
Reviewed By: simpkins
Differential Revision: D7957177
fbshipit-source-id: 7b0865a37ebbf39fdb34db409edd70d606295a0f
Summary:
I got tired of typing PathComponentPiece{"..."} in tests so here are
some operator literals.
Reviewed By: simpkins
Differential Revision: D7956732
fbshipit-source-id: 85d9f3fd725853a54da9e70fc659bd7eb9e0862c
Summary:
This moves some things around in order to facilitate adding the migration
command in a separate file.
Reviewed By: bolinfest
Differential Revision: D7946842
fbshipit-source-id: 54a554fb02e83a12f1d626b81377bc042fac41aa
Summary:
This allows using multiple cores when supported by
the BackingStore, and improves the throughput of prefetches.
Reviewed By: chadaustin
Differential Revision: D7888343
fbshipit-source-id: 1747f4ec4edf9ace02d54a4fb0ea3e8f509f51e5
Summary:
this overrides the LocalStore::getFuture to use its own
thread pool.
Reviewed By: chadaustin
Differential Revision: D7888344
fbshipit-source-id: 76b18d9417b28dc0ab72af8d070bc9e037c73bc3
Summary:
Adds a dumb `getFuture` implementation in the LocalStore
base class that simply calls `get`. Different store implementations
may choose to override this to allow making use of multiple cores
if appropriate.
Reviewed By: chadaustin
Differential Revision: D7888345
fbshipit-source-id: 20ba2db91cd7d62e5594f7d3bc3fca594dd107aa
Summary: This old Overlay code is no longer necessary.
Reviewed By: simpkins
Differential Revision: D7903912
fbshipit-source-id: 4a39d6ce7d1f6f81eb13715f2d5d17b22c10d413
