Summary:
Drop the --thrift_address flag, and always create the thrift socket using a
fixed name under the `.eden` directory. The location of the `.eden` directory
is still configurable with the `--edenDir` argument.
There isn't really much benefit to making the socket path be configurable
separately from the .eden directory path, and it adds to the code complexity.
For instance, while you can tell eden to listen on a TCP socket instead of a
Unix domain socket, that functionality has been broken since D4637285
introduced a `CHECK()` statement that crashes the code when using a TCP socket.
Reviewed By: wez
Differential Revision: D6929863
fbshipit-source-id: ee5f7341d01d3ce522cae936ef3c133bba3f18f7
Summary:
Update EdenMount::diff() to use the UserInfo object stored in the shared
ServerState rather than calling UserInfo::lookup() on each diff operation.
Reviewed By: wez
Differential Revision: D6929865
fbshipit-source-id: a68ab1fa9eb345b59972e67c3aac258b4dbcdab5
Summary:
Add a new ServerState class to store process-wide state that is shared across
multiple mounts. Up until now we have passed around the shared state data as
separate variables.
This is intentionally separate from the existing EdenServer class to allow unit
tests to create EdenMount objects without an EdenServer object.
Reviewed By: wez
Differential Revision: D6929861
fbshipit-source-id: 5f22efb6d79dfd70031be1dc37f494c2ad8af902
Summary:
We no longer need this and I believe that this was contributing
to this source of flakiness in our CI; this stack trace triggers when
we get an aborted or short read from the kernel:
```
*** Aborted at 1517827659 (Unix time, try 'date -d 1517827659') ***
*** Signal 11 (SIGSEGV) (0x2d0) received by PID 153706 (pthread TID 0x7f63ea37d700) (linux TID 160573) (maybe from PID 720, UID 0) (code: address not mapped to object), stack trace: ***
@ 0000000002abfa2d folly::symbolizer::(anonymous namespace)::signalHandler(int, siginfo_t*, void*)
./folly/experimental/symbolizer/SignalHandler.cpp:413
@ 00007f646e96dacf (unknown)
@ 00007f646e96aa20 __pthread_kill
@ 0000000001873748 facebook::eden::fusell::FuseChannel::requestSessionExit()
./eden/fs/fuse/FuseChannel.cpp:483
@ 000000000187d2f7 facebook::eden::fusell::FuseChannel::processSession()
./eden/fs/fuse/FuseChannel.cpp:656
@ 000000000187dab9 facebook::eden::fusell::FuseChannel::fuseWorkerThread(unsigned long)
./eden/fs/fuse/FuseChannel.cpp:493
@ 00007f646f267170 execute_native_thread_routine
@ 00007f646e9637a8 start_thread
@ 00007f646e047a7c __clone
```
my theory is that we're allowing shutdownImpl to free things out from under other
threads before they've all seen this signal and wound down fully. This is slightly
speculative in that I haven't managed to reproduce this stack trace on my devserver.
We don't really need this additional signal any longer.
Reviewed By: simpkins
Differential Revision: D6907734
fbshipit-source-id: 0f0138b631a7201fc9a4a1c93c2cde846e869cbd
Summary: Small refactoring I should have done with the previous diff.
Reviewed By: simpkins
Differential Revision: D6927152
fbshipit-source-id: 1dcda01134c3d63c62169c5728dba24ca0eebd68
Summary:
This enables dropping in alternative implementations
of LocalStore and adds a MemoryLocalStore implementation for
use in our tests.
This diff doesn't change the default storage option for the
eden server. I'll look at adding such an option in a follow up diff.
Reviewed By: chadaustin
Differential Revision: D6910413
fbshipit-source-id: 018bf04e0bff101e1f0ab35e8580ca2a2622e5ef
Summary:
Change `LoggerDB::get()` to a reference instead of a pointer since this
function can never return null.
Reviewed By: yfeldblum
Differential Revision: D6893206
fbshipit-source-id: af47063918a79c851fd39b838d6c63755166e033
Summary:
Update hg_import_helper.py to parse the repository's .hg/hgrc file, but then
create the repository object with a fresh UI object that has not parsed that
config yet.
This more closely mimic's the behavior of mercurial's dispatch code invoked
when starting the `hg` command line. This behavior is required to ensure that
secondary repository objects that get created end up with the correct
configuration, and do not have settings from the original repository's hgrc
file.
The mercurial behavior of parsing the original repository's hgrc file twice
dates back to rHG741f64dfc04d1.
Reviewed By: quark-zju
Differential Revision: D6909449
fbshipit-source-id: 85073ab6ade4ab70247d48bc670c9924e9e6841f
Summary:
Update `TreeInode::diff()` to check if its hash matches the source control tree
it is being compared to, and return early if they are identical.
I'm surprised that I forgot to include this initially when implementing
`TreeInode::diff()`
This makes `hg status` faster when a large number of unmodified directories
have been loaded.
Reviewed By: chadaustin
Differential Revision: D6890615
fbshipit-source-id: 561630d0220b4875dbf3678161cdb41a8aa4fc82
Summary:
This re-orders some of the code in `TreeInode::diff()` slightly. This should
not affect the behavior of the code.
This moves the `isIgnored` check inside the main `contents_.wlock()` block.
This reduces the number of places where we grab the lock, and will help keep
things simple for an upcoming diff where I need to add some more checks in this
code with the lock held.
This also changes `inodeFuture` to use the new `Future::makeEmpty()`
constructor rather than having to use an `Optional<Future>`
Reviewed By: chadaustin
Differential Revision: D6890616
fbshipit-source-id: 354bbf6a6be6d356fd23e6c0fb6b534679bbe0bb
Summary: We don't think this is happening, but let's test it!
Reviewed By: simpkins
Differential Revision: D6888038
fbshipit-source-id: 754b2ec8f78ff513fd350a74505915e2a1e9ba3e
Summary: Small things I've needed in later diffs.
Reviewed By: wez
Differential Revision: D6877755
fbshipit-source-id: c9002eb0b92dbd8fe9c4f636d2ca79b25cde331f
Summary:
While working on timestamp storage, the fact that
InodeTimestamps was a member of InodeBase kept getting in the way.
Make it its own type.
Reviewed By: simpkins
Differential Revision: D6862835
fbshipit-source-id: 91d8984764f0586b9fa52e961eb5606a530e0416
Summary: This API seems like it should be const, as it does not modify the clock.
Reviewed By: chadaustin, zhupanov
Differential Revision: D6869719
fbshipit-source-id: c8bf4ccab34538b59e6baeedd0b0ff88b328236e
Summary:
This diff moves the mount-time initialization handling
out of the main loop. This rationale for this is:
* We don't (and shouldn't!) need to process FUSE_INIT for takeover
processing, and this structure allows us to make stronger assertions
about our state.
* we can avoid spinning up multiple threads in the (rare!) case that
the FUSE_INIT fails
* It is now a little harder for exceptions during initialization to
escape our notice.
In rearranging this stuff, I found a race condition in the worker thread
shutdown; we could erroneously emit a completion event before all of
the threads had been torn down and this resulted in sporadic integration
test failures hitting the assertion for the number of joined threads
in the destructor.
Reviewed By: simpkins
Differential Revision: D6766330
fbshipit-source-id: 32afb5a7c739c75aebfdb0a8f896eec5f41ad33f
Summary:
This is the spiritual successor to D3302706 which originally
wanted to solve this by adding a python extension. That would prove
to be too painful for the opensource build so it was shelved.
We now need to be able to run our tests in an environment that doesn't
have the `attr` rpm installed so this is a good time to fix this
in a more portable way.
This diff adds a little wrapper around the functions that we already
have for consuming extended attribute information and augments them
with another to list attributes.
The utility emits output in json format and is intended to be fed
directly into the helper functions we have in `fs.py`.
Reviewed By: chadaustin
Differential Revision: D6851182
fbshipit-source-id: 3d1d1a351f2e01405645d45658d1c8bc61a659a4
Summary:
Dir's contents were represented as a vector of 64-bit
pointers to 48-byte structs. This change removes that layer of
indirection, reducing memory usage and slightly pessimizing insertion.
The diff is mostly mechanical outside of the TreeInode.h changes and
calls to emplace..
I'll run memory tests tomorrow, though it's a gamble as to whether
private bytes will show a difference. I may need to shrink the Entry
struct too.
Reviewed By: wez
Differential Revision: D6804957
fbshipit-source-id: b126656dbc7951565e74b6401adde6353e809056
Summary:
This adds version 3 nee 2 of the takeover serialization,
which uses thrift to represent the data. Even though it is logically
version 2 I'm naming it version 3 because we're taking advantage of
the MessageType values from Version 1 which allowed numerical values
1 or 2 to represent different data types.
In this diff we now formalize that first word as the protocol version
and to avoid ambiguity are starting the new version value at 3. I
did briefly consider using the name Version2 to refer to this and setting
the value in the enum to 3, but I didn't want to become known for
API hate crimes against my fellow engineers.
Reviewed By: simpkins
Differential Revision: D6733406
fbshipit-source-id: e2067365e4e8b388490440fd73ab504544011846
Summary:
Whilst chatting with simpkins we realized that we lost
the handshake portion of the takeover protocol during a refactor.
The handshake is important for a couple of reasons:
1. It prevents unmounting and loosing all the mounts in the case
that sometime decides to netcat or otherwise connect to the
socket
2. It gives us an opportunity to short circuit any heavy lifting
if we know that it will be impossible to succeed.
3. It allows us to rollback to earlier builds with older versions
of the takeover protocol.
This diff adds a little bit of machinery to enable passing a set of supported
takeover protocol version numbers. The intent is to retain support for
the two of these at a time; any time we change the encoding/protocol
for takeover we'll bump the version number and add supporting code
to handle the new format, retaining support for the prior version.
Retaining the ability to handle the prior version allows us to downgrade
to an earlier build gracefully if/when the need arises.
I opted to do this here rather than by bumping the `kProtocolID`
constant in `UnixSocket.h` becase we're not really changing the
lowest level of the protocol; just the takeover specific portions.
I haven't actually changed the takeover serialization in this diff,
but do have some work on that happening in D6733406; that diff will
be amended to take advantage and demonstrate how this versioning
scheme works.
A key thing to note about the implementation of this diff is that
the client sends the version number to the server, but doesn't
add any explicit version encoding in the response we receive.
This is deliberate and allows us to upgrade prior builds to
this new scheme. I'll add a more definitive check for this
situation when I actually rev the format in the following diff.
Reviewed By: simpkins
Differential Revision: D6743065
fbshipit-source-id: c991cebfee918daad098105ca6bcfef76374c0ff
Summary:
Tiny thing I noticed when reading code. Keep the entry name
as a StringPiece rather than bouncing through char*.
Reviewed By: zhupanov
Differential Revision: D6820080
fbshipit-source-id: 884e55f74094f44012efbe44b86d8e5903300967
Summary:
isSameAs calls getSha1 which was failing on symlinks. The
original concern was that asking for the SHA-1 of a symlink is
ambiguous: do you want the hash of the symlink or the target? But we
already check for whether you are requesting the SHA-1 of a symlink in
EdenServiceHandler, so it's redundant and incorrect to check in
FileInode too.
Reviewed By: simpkins
Differential Revision: D6847489
fbshipit-source-id: 13966da06bcde75c5c568e09fef14e735de47cfb
Summary:
In my benchmarks, using fbstring over std::string for PathMap is
worthwhile. There are a lot of entries stored in PathMap and the fact
that fbstring is a word smaller than std::string adds up.
Differential Revision: D6820204
fbshipit-source-id: 1ac34a81828891eb8125b13a074de6ed9715edad
Summary:
Added comments linking to the task with discussion of why we are postponing
proper handling of umask and permissions in general for mkdir and mknod.
Reviewed By: wez
Differential Revision: D6808696
fbshipit-source-id: a6d74ea08448c8578282dc146ac81da187d4409c
Summary:
Now that we've moved the hg source around, update eden
to point to the source of truth rather than the synced subdir.
I spoke with andrewjcg about the use of header_namespace here;
it is required to be able to reference the headers without changing
the code under scm/hg to be relative to the fbcode root. We can't
change that code in that way because it has to build standalone
for the opensource consumers.
Remove the duplicated code while we're in here to make sure
that we're not accidentally referencing it anywhere.
Reviewed By: simpkins
Differential Revision: D6792969
fbshipit-source-id: ec55e98c0bde2488c213b1fd9f8baae9868a350c
Summary:
It's interesting to see the total number of loaded files
vs. trees when the loaded inode count is high.
Reviewed By: wez
Differential Revision: D6765874
fbshipit-source-id: 178b30184428bd5cf5e005eb475e4f5a1476c385
Summary:
I'm not sure what was wrong with the old code, but I
simplified and clarified all of the time math and now `eden debug
unload` behaves as I'd expect it should.
Reviewed By: simpkins
Differential Revision: D6764962
fbshipit-source-id: 3ed359d4ab4652e95d1538a0982c24185999351c
Summary:
rdev doesn't add any value yet. We can add it back if we want
to implement support.
Reviewed By: simpkins
Differential Revision: D6792346
fbshipit-source-id: ce16317074f1daa456737c55804da8fb7f2b7a94
Summary: This makes it easier to separate out the fuse initialization logic in a follow on diff.
Reviewed By: simpkins
Differential Revision: D6766329
fbshipit-source-id: 178d97e2302bc283c701039494e4477285abc0af
Summary:
This slightly improves the code in EdenMount by having
just a single call to the FuseChannel constructor with an optional
`fuse_init_out` instance that we can use to determine if we are
performing a takeover or not.
Reviewed By: simpkins
Differential Revision: D6766328
fbshipit-source-id: ece140a1572e2934a3e35bfe25b83af910346c18
Summary: Minor stylistic changes that were done during constification but factored out.
Reviewed By: chadaustin
Differential Revision: D6774976
fbshipit-source-id: d18cd339153cf16ff69be0de5f3eb019a4baa1a0
Summary:
The value we get is of enum class, so it is guaranteed to be in the valid set.
Omitting default we get compiler support if we ever expand the set of values.
Throw becomes obviated, because it becomes unreachable.
Reviewed By: simpkins
Differential Revision: D6771224
fbshipit-source-id: 039f0545d20649193b3fb74981ee7f027f31fd8e
Summary: Replaced memset used to 0-init structs with POD fields with = {} init.
Reviewed By: simpkins
Differential Revision: D6770868
fbshipit-source-id: 190205a277e3c73711813bc515b860b107d41f83
Summary:
Add a log message on successful shutdown, just so that Eden's shutdown point is
easily identifiable in the log files.
Reviewed By: wez
Differential Revision: D6757386
fbshipit-source-id: f19a37c2ec9ae64fff99d40201d80e5f8faeaea5
Summary:
Update the eden hg extension to check for the remotefilenames extension both as
`remotefilenames` and as `hgext.remotefilenames`.
Reviewed By: bolinfest, quark-zju
Differential Revision: D6755872
fbshipit-source-id: 261e473c7194d67c69e1a008bf16d563a87fba67
Summary:
akushner mentioned that he saw a hang when performing
a graceful restart and we realized that his cwd was inside the
mount point that he was taking over. While this doesn't guarantee
problems (I've run it this way almost exclusively with no issues)
it feels like a potential problem to hold a reference to a file
descriptor within a fuse mount that we are ourselves going to
manage: it seems likely that we could end up deadlocking if
we were to attempt any IO on such a descriptor.
This diff seeks to head off this particular case by simply
chdiring to `/` very early on startup. This is typically
something that is handled by the `daemonize` function in
old school unix code.
Reviewed By: simpkins
Differential Revision: D6740547
fbshipit-source-id: 986004e9ed6cfa493ad55c5cdbca1805fc0f7768
Summary:
This is moving some files around in preparation for
moving TakeoverData to using thrift for its serialization
Reviewed By: simpkins
Differential Revision: D6733405
fbshipit-source-id: 235ba237546f8ef606de8445db45683ce38a2d2c
Summary: Minor refactor replacing if with switch and a loop control comparison.
Reviewed By: wez
Differential Revision: D6730849
fbshipit-source-id: 69ac7be9fae3be6274b56dd2baea433ed2437f7c
Summary:
This implements a TODO/FATAL that is important for
graceful restarts to be useful in my "acid test" scenario,
which is to perform a graceful restart while buck build is
running.
Reviewed By: simpkins
Differential Revision: D6700189
fbshipit-source-id: dec1b818ebc9e907841bc127ee08c953b59d6487
Summary:
Update the statfs() code to return non-zero values for namelen and frsize.
Returning 0 for namelen was causing problems for programs that checked
`pathconf(path, _PC_NAME_MAX)` and tried to honor this value. For instance,
GNU patch would try to generate files with a 0-length name since we indicated
the maximum name length was 0.
I haven't investigated too closely, but this behavior might have broken only
recently when we stopped using libfuse. libfuse may have been setting this to
255 for us. I didn't see code in libfuse that would do this, but I'm fairly
sure GNU patch was working correctly very recently.
Reviewed By: wez
Differential Revision: D6710370
fbshipit-source-id: fc9a0320cd7c1eb2545219a3ec123c2f0644fb5d
Summary:
This avoids the long delay and the `Cpp2Worker.cpp:281] Failed to join
outstanding requests.` message from showing up in the eden logs during shutdown
and graceful restart.
Reviewed By: chadaustin
Differential Revision: D6693117
fbshipit-source-id: 3c56314f288a596264cddae0d8bbab66ab19e9fe
Summary:
Previously, `EdenMount::create` would implicitly call
`EdenMount::initialize` which would load the root inode and the `.eden` magical
directory. That's fine for the fresh mount case, but in the case of the
graceful restart we need to take the opportunity to apply the `InodeMap`
from the old process before we start muddying its state.
This diff breaks out the `initialize` method from the `create` method and
makes the mount code call it after potentially loading the `InodeMap` from
the takeover data.
In addition, this diff removes the the `root->loadMaterializedChildren()`
call from the mount initialization code. It is no longer required to do
this eagerly and it makes things simpler and our memory profile a little
smaller to defer this (I haven't measured how much that impacts things).
Reviewed By: simpkins
Differential Revision: D6691182
fbshipit-source-id: 52033a6d64105b658314a919f69dbfcd4eea242b
Summary:
This removes the duplicated logic and makes it a little
bit easier to follow the initialization sequence.
It doesn't change the behavior, just moves some code around.
Reviewed By: simpkins
Differential Revision: D6691180
fbshipit-source-id: 2068dbe56ebe9a6136d69689997aec8dedd32be0
Summary:
This is the "simple" threading through of a `doTakeover`
flag and the return of `Optional<TakeoverData>` in the unmount code.
Reviewed By: simpkins
Differential Revision: D6691181
fbshipit-source-id: 4a384787783c16085f2e9964964023ba07cefca3
Summary:
I'm so-so on a bit of the implementation here, but it works!
I had to change the `takeoverPromise` from the `pair<fuseDevice, connInfo>`
to a new helper struct because we now have three distinct pieces of data
to pass out of EdenMount to build up the overall TakeoverData.
The key change in this diff is that we have to release all of the file handles
we're maintaining in the `FileHandleMap` prior to shutting down the `InodeMap`,
otherwise the `InodeMap` will never complete (it waits for all inodes to be
unreferenced, and that cannot happen while there are open file handles). I've
made the `FileHandleMap` serialization and clearing contingent on performing a
takeover shutdown because that feels like the safest thing to do wrt. not
losing any pending writes.
Reviewed By: simpkins
Differential Revision: D6672437
fbshipit-source-id: 7b1f0f8e7ff09dbed850c7737383ecdf1e5ff0c7
Summary:
This is the key portion that makes the graceful restart
function. This diff connects almost all of the moving pieces together;
it informs the priv helper about the takeover mount and transfers
the InodeMap information into the new generation of the eden server.
It doesn't yet load the fileHandleMap (will tackle that in a follow up diff)
Reviewed By: simpkins
Differential Revision: D6670903
fbshipit-source-id: 1770d99eb1477440a6c1deed83b0da55b9c1bbe4
