Summary:
Add a method to LogHandler to return its current configuration. This will
make it possible to query the LoggerDB for its current configuration state.
Reviewed By: bolinfest
Differential Revision: D6200563
fbshipit-source-id: 2b8b9752bbeb26c8aac28d1a73b7e2312fd198c8
Summary:
I accidentally broke the flatmanifest fallback code in D6333613 by changing the
exception type thrown for errors received from hg_import_helper.py but not
updating the catch statement in HgImporter::importTreeImpl().
This updates importTreeImpl() to catch the new HgImportPyError type correctly.
I have dropped the check on the error message entirely, since the mercurial
python code can throw a variety of errors that all mean this tree data isn't
available.
Reviewed By: bolinfest
Differential Revision: D6434359
fbshipit-source-id: c62d3c1667681712293873de2b9bf6d9220da767
Summary:
Increase the eden daemon start timeout from 5 to 60 seconds.
With the 5 second timeout I sometimes see `buck test eden/...` time out in the
clone integration test which waits for eden to start. This likely takes longer
than normal when lots of other integration tests are also running in parallel.
Even when simply manually starting eden I have seen it take nearly 60 seconds
to open the RocksDB database. (I suspect RocksDB performs some extra checking
when opening a DB created by an older version of the code.) Eden requires the
RocksDB be open before it starts responding successfully to health check
requests.
Reviewed By: wez
Differential Revision: D6434357
fbshipit-source-id: 5f62ff821ecd64f7a1345e611f2299177513e411
Summary: Update the functions in cli/util.py with type annotations.
Reviewed By: bolinfest
Differential Revision: D6434356
fbshipit-source-id: 054d81427132229a390b8a133d5180be5f70bd20
Summary: Annotate more integration test functions with type information.
Reviewed By: bolinfest
Differential Revision: D6434358
fbshipit-source-id: b88351eebee58561465752378c6771b7b1f9554e
Summary:
This fixes a crash in EdenMount::destroy() if EdenMount::create() failed to
load the root inode. Previously the code called shutdownImpl() in this case
which tried to unload all inodes and crashed since the root inode was null.
This also fixes EdenMount::destroy() to properly handle the FUSE_ERROR and
FUSE_DONE cases.
Reviewed By: wez
Differential Revision: D6434355
fbshipit-source-id: 39c5f4472d6ebbcf881b4c9c8c8fd67686032ec1
Summary:
There's a bug in some combination of Eden and FUSE where open(O_TRUNC)
followed by a sequence of writes over an existing file does not flush
the kernel's VFS page cache, which manifests as an mmap larger than
the file's size not zeroing the data beyond the file's size. These
tests attempt capture that use case, but they are fiddly.
Disabling ATOMIC_O_TRUNC seems to resolve the issue.
Reviewed By: wez
Differential Revision: D6430152
fbshipit-source-id: f7626e268e778ebab60c66322e0ce42bce746ae1
Summary:
The goal is to provide a fast path for watchman to flesh
out the total set of changed files when it needs relay that information
on to consumers.
We choose not to include the full list in the Journal when checking out
between revisions because it will not always be needed and may be an
expensive `O(repo)` operation to compute. This means that watchman
needs to expand that information for itself, and that is currently
a fairly slow query to invoke through mercurial.
Since watchman is responding to journal events from eden we know that
we have tree data for the old and new hashes and thus we should be
able to efficiently compute that diff.
This implementation is slightly awful because it will instantiate an
unlinked TreeInode object for one side of the query, and will in
turn populate any children that differ as it walks down the tree.
A follow on diff will look at making a flavor of the diff code that
can diff raw Tree objects instead.
Reviewed By: bolinfest
Differential Revision: D6305844
fbshipit-source-id: 7506c9ba1f4febebcdc283c414261810a3951588
Summary:
In onboarding users, we usually tell them to run `eden clone fbsource`,
but that fails because users generally have not run `eden daemon` yet.
The simplest thing is to do it for them when they run `eden clone` when
the daemon is not running.
Reviewed By: wez
Differential Revision: D6357249
fbshipit-source-id: dc112c1efe214485e3c5c8e06522d299a100d3a0
Summary:
This should have been removed as part of D6179950.
Ideally, Buck would error when this happens, but apparently `glob()` does not
complain when patterns do not match any files, even when the pattern does not
contain any wildcards. There appears to be some code at Facebook that is
exploiting this behavior.
Reviewed By: simpkins
Differential Revision: D6421529
fbshipit-source-id: c6f982624e0e12a911bc12ab1e8239ba4358ea56
Summary:
This corrects a bug where the in-memory timestamp for new files would
be set to the last checkout time instead of when the file was created.
Reviewed By: simpkins
Differential Revision: D6366189
fbshipit-source-id: c5fa8c779726d3a75c2d57b2a161293297eb9271
Summary:
Per a discussion with wez, let's make it explicit in the comments that the kernel handles
O_EXCL for us.
Reviewed By: simpkins
Differential Revision: D6365157
fbshipit-source-id: 638f67cd89a9450ff084e0ef77c731ef738bf518
Summary:
In some workloads we're seeing folks run out of file descriptors.
We forgot that we'd taken out the code that closes the underlying fds.
This diff takes a run at adding a simple counter of the open file handle
objects that is incremented when they are constructed and decremented
when they are destroyed.
When the count falls to zero we release the file handle.
Note that we unconditionally open files when we first load the inodes
from the overlay. I tried to defer that open attempt and it broke
the timestamp overlay test. I think we can revisit that aspect in
a follow on diff; for now we should be more resilient to transiently
opened files from things like ripgrep or similar.
Reviewed By: simpkins
Differential Revision: D6097090
fbshipit-source-id: 9a48220002e760fb1ffb8d7e2a68fa7036558b78
Summary: It is currently unused. Let's bring it back if/when we need it.
Reviewed By: chadaustin
Differential Revision: D6368867
fbshipit-source-id: 096015ba597a6e04f544273ba9773576429e39ce
Summary:
This bug is part of a bigger issue in our Mercurial integration where
`UNTRACKED_ADDED` conflicts are being silently swallowed in our Hg extension
whereas stock Mercurial presents these as conflicts and forces the user to deal
with them. The Mercurial issues will be addressed in a follow-up change.
Reviewed By: simpkins
Differential Revision: D6365580
fbshipit-source-id: 831e27ce1da90ea605033b2b9988fe400ba404aa
Summary:
We now run two versions of this test: one where the file that exists in the
destination commit is untracked before the update and one in which it is added
before the update.
Reviewed By: simpkins
Differential Revision: D6334002
fbshipit-source-id: ef6bffa27bc18171b5e21dc284c7a21aa6e35da4
Summary: Addresses outstanding TODO now that D6322052 has landed.
Reviewed By: simpkins
Differential Revision: D6368884
fbshipit-source-id: 497c42466e05af0f1690bc6401b1d271de691e58
Summary:
In preparation for bringing D6097090 across the finish line, we need a
more explicit mechanism for determining which state the inode is in.
The issue is that whether an inode is materialized was checked in a
couple ways: the nonexistence of the hash field or the definedness of
the file field. This diff introduces an explicit enum indicating the
state.
Reviewed By: simpkins
Differential Revision: D6325955
fbshipit-source-id: 3682a4ebc9330193baadbb33a4dd9845f26e59a6
Summary:
Define constants at the top of EdenServer.cpp for the names of the main lock
file, the thrift socket, and the takeover socket.
Reviewed By: bolinfest
Differential Revision: D6295040
fbshipit-source-id: 8605840a50c84bc89b798123d1063bbb11ff2502
Summary:
This begins implementing the "client-side" portion of graceful takeover in
edenfs. When the --takeover flag is specified, if edenfs finds that another
edenfs process is already running it will attempt to gracefully take over its
mount points rather than exiting with an error.
This does not yet actually take over the mount points themselves--it still
sends dummy mount information during shutdown, and does not use this data
during startup. However, we do perform takeover of the eden lock file and the
thrift server socket.
Reviewed By: bolinfest
Differential Revision: D6038944
fbshipit-source-id: 42406a0559367cec79af088b4ca84c22de3f3ef3
Summary:
Update the TakeoverData to also include the thrift server socket.
Also update EdenServer to set this field when performing a takeover
shutdown.
Reviewed By: bolinfest
Differential Revision: D6038945
fbshipit-source-id: 725faa431b3b55d617ef645c8a7eae080e4fe066
Summary:
Update EdenServer to implement the TakeoverHandler API, and to exit after
sending the mount point takeover data. The actual shutdown logic itself is not
implemented yet--this just sends dummy data for now. However, this does serve
as a proof of concept that the TakeoverServer and TakeoverClient code functions
as desired.
Reviewed By: bolinfest
Differential Revision: D6018180
fbshipit-source-id: c19581928926a46b767f1ee5c1761381e5055fa9
Summary:
This adds a new class which listens on a Unix domain socket for clients that
wish to gracefully take over Eden's FUSE mount points. The goal is to
eventually enable graceful restart functionality for eden.
It would be nice if we could use the existing thrift server socket for this,
but thrift doesn't provide low-enough level APIs so that we can send
credentials and file descriptors over the socket using SCM_CREDENTIALS and
SCM_RIGHTS. Using our own separate socket is the easiest way to accomplish
this instead.
For now eden just listens on this socket and logs a message when a client
connects; this diff does not yet contain logic for performing mount point
takeover.
Reviewed By: bolinfest
Differential Revision: D5827752
fbshipit-source-id: 928e541efa2546cb612da2699ff0bd822bafaad5
Summary:
Eden attempts to initialize timestamps of newly loaded inodes with the time of
the last checkout operation performed in this mount. Unfortunately it had a
number of bugs in this logic:
EdenMount had two separate fields attempting to track the last checkout time:
`lastCheckoutTime_` and `parentInfo_.lastCheckoutTime`.
Unfortunately neither field was actually updated on checkout operations.
Additionally, `lastCheckoutTime_` did not have any locking to allow it to be
updated. `parentInfo_.lastCheckoutTime` did have locking, but it used the
mount point's checkout lock, so it could not be accessed during checkout
operations.
This diff removes `parentInfo_.lastCheckoutTime`, keeping only
`lastCheckoutTime_`. It also converts `lastCheckoutTime_` to a
`struct timespec` since this is most often needed as a `timespec`. It also
adds a new mountpoint-wide lock for synchronizing accessing to this variable.
Reviewed By: bolinfest
Differential Revision: D6356698
fbshipit-source-id: db54f9bb297b5febe4642e2b3fcc8055a6afc199
Summary:
This fixes up a minor but sadly fatal oversight from D6314115; a call to
`get_repo_config` didn't get renamed to `find_config_for_alias`
(Note: this ignores all push blocking failures!)
Reviewed By: bolinfest
Differential Revision: D6354474
fbshipit-source-id: 13c665882419feebf0b8c0596eb3e0220ee2cf13
Summary:
It seemed unnecessary to include a `\n` in the template only to strip it.
Similarly, we should be strict when parsing the output since we expect it to be
a string of hex characters.
Reviewed By: wez
Differential Revision: D6322129
fbshipit-source-id: 61c483badfd7b68ed012310360aa582d6bdf5181
Summary:
When cloning an existing Eden mount, we should be smart and inherit its
underlying config so that we inherit properties such as its bind mounts.
Reviewed By: wez
Differential Revision: D6322002
fbshipit-source-id: 3f5ba135b12ad7dcecef6676d27495cfbf0ce97b
Summary:
Previously, a user had to define a config for a repo in a file like `~/.edenrc`
in order to create a new Eden mount via `eden clone`. In practice, the
information that is hardcoded in the config can generally be inferred from an
existing repo, so this expands `eden clone` to support both modes of operation.
Note this made it possible to finally unify the `RepoConfig` and `ClientConfig`
types. This revision removes `RepoConfig`, so I dutifully renamed every
local variable named `repo_config` to `client_config`.
Reviewed By: wez
Differential Revision: D6314115
fbshipit-source-id: 9625a5fbe35b30f76b6099180580c64435a4cf72
Summary:
As explained by the comment in the file, the alias for the config that was used
to create the client should not be necessary to mount it. This builds on top of
D6310325 to rewrite some internal functions that still relied on this alias
being present.
Reviewed By: wez
Differential Revision: D6313778
fbshipit-source-id: 867300e9ec844731993376b8957935f175a242f5
Summary:
The headline changes of this revision are:
- Changes the format of the config file from INI to TOML
(the `edenrc` file under `~/local/.eden` has been replaced
with `config.toml`). This revision includes logic for automatically
performing the migration when Eden is restarted.
- Inlines data from `/etc/eden/config.d` into the TOML file.
Historically, the `edenrc` file for a client would contain the
name of the "configuration alias" defined in a config file like
`~/.edenrc` or `/etc/eden/config.d/00-defaults`. When Eden
loaded a client, it would have to first read the `edenrc` and
then reconstitute the rest of the client configuration by
looking up the alias in the set of config files that were used to
create the client in the first place.
This changes things so that all of the data that was being
cross-referenced is now inlined in the client's config file.
This makes loading a config considerably simpler at the cost
of no longer being able to change the config for multiple clients
that were cloned from the same configuration alias in one place.
It was questionable whether being able to modify a client from
a foreign config after it was created was a safe thing to do, anyway.
Eliminating the need for a historic link to the configuration alias
will make it easier to support running `eden clone` on an arbitrary
local Hg or Git repo. So long as `eden clone` can extract enough
information from the local repo to create an appropriate config file
for the new Eden client, there is no need for a configuration alias
to exist a priori.
Since we were already changing the data in the config file, this
seemed like an appropriate time to make the switch from INI to
TOML, as this was something we wanted to do, anyway.
In testing, I discovered a discrepancy between how boost's
`boost::property_tree::ptree` and Python's `ConfigParser` handled
the following section heading:
```
[repository ZtmpZsillyZeden-clone.LIkh32]
```
Apparently `hasSection("repository ZtmpZsillyZeden-clone.LIkh32")`
in boost would fail to find this section. Because
[[https://stackoverflow.com/questions/13109506/are-hyphens-allowed-in-section-definitions-in-ini-files | there is no spec for INI]],
it is not that surprising that boost and `ConfigParser` do not 100% agree
on what they accept. Moving to TOML means we have a configuration
language with the following desirable properties:
- It has a formal spec, unlike INI. This is important because there are parsers
in a wide range of programming languages that, in theory, accept a consistent
input language.
- It is reasonable for humans to write, as it supports comments, unlike JSON.
- It supports nested structures, like maps and arrays, without going crazy
on the input language it supports, unlike YAML.
Eden now depends on the following third-party TOML parsers:
* C++ https://github.com/skystrife/cpptoml
* Python https://github.com/uiri/toml
This revision also changes the organization of `~/local/.eden` slightly. For now,
there is still a `config.json` file, but the values are no longer hashes of the realpath
of the mount. Instead, we take the basename of the realpath and use that as the
name of the directory under `~/local/.eden/clients`. If there is a naming collision, we
add the first available integral suffix. Using the basename makes it easier to
navigate the `~/local/.eden/clients` directory.
Although the `edenrc` file under `~/local/.eden/clients` has been switched from INI
to TOML, the other Eden config files (`~/.edenrc` and `/etc/eden/config.d/*`) still use
INI. Migrating those to TOML will be done in a future revision.
Note this revision allowed us to eliminate `facebook::eden::InterpolatedPropertyTree`
as well as a number of uses of boost due to the elimination of
`ClientConfig::loadConfigData()` in the C++ code. Because `ClientConfig`
no longer does interpolation, a bit of `ClientConfigTest` was deleted as part of
this revision because it is no longer relevant.
Reviewed By: wez
Differential Revision: D6310325
fbshipit-source-id: 2548149c064cdf8e78a3b3ce6fe667ff70f94f84
Summary: More work towards encapsulating a FileInode's internal state machine.
Reviewed By: wez
Differential Revision: D6316013
fbshipit-source-id: 9c8303b35a0de1ba69207c7f59be88c5fb037ad8
Summary:
This flips the sense of the unmount command; previously
we would default to destroying the mount and associated state,
but this was a bit of a massive sharp edge to our UX.
Now the default is non-destructive and you have to explicitly
pass in `--destroy` to enable destructive mode.
Reviewed By: chadaustin
Differential Revision: D6346013
fbshipit-source-id: ce612e7d8a70540d63217a97f96bc5760f3951af
Summary:
This is preparation for a change I'm preparing that makes the state
transitions in FileInode clearer and safer.
Reviewed By: wez
Differential Revision: D6304272
fbshipit-source-id: 493ee80517443432f790abf9806000eecb03651c
Summary:
The messaging implies that something is already mounted, especially
if one is used to the error messages that are emitted by `mount`. In this
situation we're complaining that we already have a repo configured at that
path, so let's help point folks in the right direction in the error message.
Reviewed By: chadaustin
Differential Revision: D6344017
fbshipit-source-id: d48176f5806b3233e191567a840c88af101002cf
Summary: Otherwise these will linger forever in the eden server
Reviewed By: DurhamG
Differential Revision: D6099753
fbshipit-source-id: 77b975dcde28cd7c3d4ae2302bddb625682d1994
Summary:
Update hg_import_helper.py to include the exception type name in error
responses. Add a new HgImportPyError class in the C++ code to include both the
python exception type name and the message string.
In the future this will give us the ability to perform special handling based
on the python exception type, rather than just on the message contents.
Reviewed By: bolinfest
Differential Revision: D6333613
fbshipit-source-id: 1074bbf9fa25ee8b1abeadc38b1a4f569bc18d13
Summary:
The gtest macros in this file were moved to folly/test/TestUtils.h
Update everything to just use folly/test/TestUtils.h directly.
Reviewed By: chadaustin
Differential Revision: D6301759
fbshipit-source-id: 7f2841c12d5bea15376f782fb3bf3bfef16039c7
Summary:
Add EXPECT_THROW_RE() and EXPECT_THROW_ERRNO() macros to folly/test/TestUtils.h
These allow more precise checks than the basic EXPECT_THROW() macro provided as
part of gtest.
These macros are being moved into folly from Facebook's eden repository
(https://github.com/facebookexperimental/eden)
This will allow us to use them in folly tests and in other projects that depend
on folly.
Reviewed By: yfeldblum
Differential Revision: D6301760
fbshipit-source-id: 1f434fb5bc9b7859f763171264fb0b2e1b4bda62
Summary:
This script automates the process of force unmounting all edenfs
mounts. It's very helpful when a `buck test eden/...` goes awry. :)
Reviewed By: wez
Differential Revision: D6314547
fbshipit-source-id: a2374c4f3220ae663a34b5ff252bf85ddab6d0aa
Summary:
Have HgBackingStore hold multiple HgImporters in its own thread pool. Incoming
requests are processed by threads in the thread pool.
Reviewed By: wez
Differential Revision: D6265043
fbshipit-source-id: b2d4f345b772f296c5335a7fbcadfce1d93245fd
Summary:
Users often run `hg update --clean .` to get out of a bad state, but this was
not clearing the "added" or "removed" state in the dirstate as it should in
Eden.
Reviewed By: wez
Differential Revision: D6331858
fbshipit-source-id: 616f187930587a1af40a1f151e3a424d50dd8da3
Summary:
https://phab.mercurial-scm.org/D1268 changed the error message that the
treemanifest code throws when it cannot fetch tree data from the remote server.
This unfortunately is thrown as a generic `error.Abort()` so eden has to match
on the error message to tell if this was an error fetching the tree vs
something else. Changing the error message broke eden's detection of this.
This updates the string that eden searches for. In the future it would
probably be nicer to make the treemanifest code throw a different exception
type so we can detect this from the exception type rather than the error
message.
Reviewed By: wez
Differential Revision: D6332802
fbshipit-source-id: 629097914e6f783729adcd25f2fd5b6d96886ab4
Summary:
Add a new debugging flag to the hg_import_helper.py script to have it just show
the manifest node ID for a given revision and then exit.
This is information is useful when debugging import issues, and I'm not aware
of an easy way to get it from the normal hg command line.
Reviewed By: wez
Differential Revision: D6332803
fbshipit-source-id: 600feac105f277d398325d2e3c50eae3d5200940
Summary:
Override dirstate.rebuild() so that it drops all file changes without trying to
mark every file in the commit manifest as normal. We don't want to track
normal files in the eden dirstate.
Reviewed By: bolinfest
Differential Revision: D6322227
fbshipit-source-id: d81ade1cdafb5fa03c642239b0cff91308c7fc35
