Summary:
Several parts of the mercurial remotefilelog and treemanifest code can
incorrectly throw KeyErrors when accessing legitimate objects that do exist.
This appears to occur if the data was created since the mercurial code on the
server side started.
These KeyErrors are often generated on the server-side of a pull or
prefetchtrees operation. We can't workaround these by invalidating caches on
our local repository object. The only way that I've been able to work around
these bogus KeyErrors is by completely recreating the repo object before
retrying.
Our code now generally attempts to load data 3 times: if the initial attempt
fails we call `repo.invalidate()` and then retry. If that retry fails we then
try destroying and recreating the entire repo object and then try again.
Ideally it would be nicer to fix mercurial itself not to throw KeyErrors for
data that does exist. However that seems somewhat more complicated.
this_is_fine
Reviewed By: wez
Differential Revision: D6986220
fbshipit-source-id: 57905dd25e11c4858822020b44185a6f83ecd363
Summary:
Explicitly mark the UnionDatapackStore as needing a refresh after we have
fetched data from the server. Otherwise we may not be able to find the tree
data that we just fetched.
Reviewed By: wez
Differential Revision: D6986219
fbshipit-source-id: c50b92ee4242665c7f5770f87a5dbab17698b8b9
Summary:
Recent changes to the treemanifest extension removed the
treemanifest.prefetchtrees() function, and replaced moved this functionality to
a repo.prefetchtrees() function instead.
Update the Eden extension to call `repo.prefetchtrees()` if it is available.
Reviewed By: wez
Differential Revision: D6986218
fbshipit-source-id: 02e07ab5194f311e49d8ecb4755d79d55acb080c
Summary:
Another small refactoring that I don't want mixed into my
bigger diffs.
Reviewed By: wez
Differential Revision: D6927482
fbshipit-source-id: 28cc60dfdffb50921a5ef9cca4e2814b90d3b701
Summary:
Introduces a persistent non-durable storage mechanism backed
by a memory-mapped file with fixed-length records.
Reviewed By: simpkins
Differential Revision: D6877217
fbshipit-source-id: 0ddacb4137cfe43e67c822dce4064356cdf515b5
Summary:
There were places we were acquiring a lock unnecessarily. In
addition, I'm looking at reducing the number of places where we store
the full mode_t to see if we can get away with dirtype_t or something
similar.
Reviewed By: wez
Differential Revision: D6972140
fbshipit-source-id: bb29a4473f3056e39596600d22e67374ca484735
Summary:
We would immediately satisfy the health check and
tell the user that the system is healthy and show the pid of
the prior incarnation rather than the one for the instance
that we just launched.
This diff refactors the health checking code so that we can
share the implementation between the cli and the integration
tests; the integration tests already had code to do the right
thing for this.
Reviewed By: simpkins
Differential Revision: D6944989
fbshipit-source-id: 7c0f02c875b1b81f8f1b7521add67928200b27ed
Summary:
chadaustin is going to think about how to test this
in unit tests intestead :-p
Reviewed By: chadaustin
Differential Revision: D6951788
fbshipit-source-id: 137f285f3a1f080ce43392a621c73640ce3a9bf7
Summary:
This makes the default `memory` for speed and minimal
flakiness, but allows a test to select a different engine where
appropriate (eg: restart and remount tests).
Reviewed By: chadaustin
Differential Revision: D6944207
fbshipit-source-id: 1fb11387beda02d059a796dad5a42d56ddcf6e88
Summary:
refactor this test so that we can apply it to any LocalStore
implementation, and have it run against the Memory, Rocks and SQLite
implementations.
Reviewed By: chadaustin, simpkins
Differential Revision: D6919455
fbshipit-source-id: cc93042b95833b175955e6395c84cf41238a90d2
Summary:
Adds a SQLite storage implementation and makes it the
default engine for integration tests; this requires fewer resources
to run and the integration tests thus run faster and more reliably.
In the future we may add a configuration option to remember the
storage engine that was used as it is currently not "safe" to switch
from one to the other because the hgproxyhash data cannot be
recreated without re-importing a revision.
Reviewed By: simpkins
Differential Revision: D6919456
fbshipit-source-id: 3afbfafb190cca0e3c797cd9b7cd051768575a8c
Summary:
Update FileInode so that getattr() and setattr() both return a reasonable value
in st_blocks.
Previously we always returned 0 in st_blocks, which caused applications like
`du` to always report files as using 0 space in Eden mounts. Now we compute
st_blocks based on st_size, so that `du` will report reasonable estimates for
when scanning the size of subdirectories inside an Eden mount.
Reviewed By: chadaustin
Differential Revision: D6932098
fbshipit-source-id: bd29e46821176e510f420e6e2b6ce480b80d50ff
Summary:
Rename the existing `state_` variable to `runState_` to help distinguish it
from the new `serverState_` variable.
The information in `runState_` is all related to whether the EdenServer object
is starting, shutting down, or running normally.
Reviewed By: wez
Differential Revision: D6929864
fbshipit-source-id: ad7af381a8a291b12db9308668c7616ebd9b7f39
Summary:
Move all of the privhelper functionality into a PrivHelper class. The
ServerState object now stores the PrivHelper object to use, rather than having
a global singleton.
This will make it easier to stub out the PrivHelper functionality during unit
tests.
Reviewed By: wez
Differential Revision: D6929862
fbshipit-source-id: e3edcb0a03ba9afdf34554cb961fd74557cdd6e3
Summary:
Drop the --thrift_address flag, and always create the thrift socket using a
fixed name under the `.eden` directory. The location of the `.eden` directory
is still configurable with the `--edenDir` argument.
There isn't really much benefit to making the socket path be configurable
separately from the .eden directory path, and it adds to the code complexity.
For instance, while you can tell eden to listen on a TCP socket instead of a
Unix domain socket, that functionality has been broken since D4637285
introduced a `CHECK()` statement that crashes the code when using a TCP socket.
Reviewed By: wez
Differential Revision: D6929863
fbshipit-source-id: ee5f7341d01d3ce522cae936ef3c133bba3f18f7
Summary:
Update EdenMount::diff() to use the UserInfo object stored in the shared
ServerState rather than calling UserInfo::lookup() on each diff operation.
Reviewed By: wez
Differential Revision: D6929865
fbshipit-source-id: a68ab1fa9eb345b59972e67c3aac258b4dbcdab5
Summary:
Add a new ServerState class to store process-wide state that is shared across
multiple mounts. Up until now we have passed around the shared state data as
separate variables.
This is intentionally separate from the existing EdenServer class to allow unit
tests to create EdenMount objects without an EdenServer object.
Reviewed By: wez
Differential Revision: D6929861
fbshipit-source-id: 5f22efb6d79dfd70031be1dc37f494c2ad8af902
Summary:
To me, restart implied shutting down the daemon and restarting it
again. Perhaps instead of `eden daemon --takeover` we should have
`eden restart`. But if people typed `eden restart` I imagine they're
trying to debug a problem, so that's probably not the right verb.
Reviewed By: wez
Differential Revision: D6929166
fbshipit-source-id: d568a1940d67f755e4c3656098c58fc81e0a3156
Summary:
We no longer need this and I believe that this was contributing
to this source of flakiness in our CI; this stack trace triggers when
we get an aborted or short read from the kernel:
```
*** Aborted at 1517827659 (Unix time, try 'date -d 1517827659') ***
*** Signal 11 (SIGSEGV) (0x2d0) received by PID 153706 (pthread TID 0x7f63ea37d700) (linux TID 160573) (maybe from PID 720, UID 0) (code: address not mapped to object), stack trace: ***
@ 0000000002abfa2d folly::symbolizer::(anonymous namespace)::signalHandler(int, siginfo_t*, void*)
./folly/experimental/symbolizer/SignalHandler.cpp:413
@ 00007f646e96dacf (unknown)
@ 00007f646e96aa20 __pthread_kill
@ 0000000001873748 facebook::eden::fusell::FuseChannel::requestSessionExit()
./eden/fs/fuse/FuseChannel.cpp:483
@ 000000000187d2f7 facebook::eden::fusell::FuseChannel::processSession()
./eden/fs/fuse/FuseChannel.cpp:656
@ 000000000187dab9 facebook::eden::fusell::FuseChannel::fuseWorkerThread(unsigned long)
./eden/fs/fuse/FuseChannel.cpp:493
@ 00007f646f267170 execute_native_thread_routine
@ 00007f646e9637a8 start_thread
@ 00007f646e047a7c __clone
```
my theory is that we're allowing shutdownImpl to free things out from under other
threads before they've all seen this signal and wound down fully. This is slightly
speculative in that I haven't managed to reproduce this stack trace on my devserver.
We don't really need this additional signal any longer.
Reviewed By: simpkins
Differential Revision: D6907734
fbshipit-source-id: 0f0138b631a7201fc9a4a1c93c2cde846e869cbd
Summary: Small refactoring I should have done with the previous diff.
Reviewed By: simpkins
Differential Revision: D6927152
fbshipit-source-id: 1dcda01134c3d63c62169c5728dba24ca0eebd68
Summary:
This enables dropping in alternative implementations
of LocalStore and adds a MemoryLocalStore implementation for
use in our tests.
This diff doesn't change the default storage option for the
eden server. I'll look at adding such an option in a follow up diff.
Reviewed By: chadaustin
Differential Revision: D6910413
fbshipit-source-id: 018bf04e0bff101e1f0ab35e8580ca2a2622e5ef
Summary:
Change `LoggerDB::get()` to a reference instead of a pointer since this
function can never return null.
Reviewed By: yfeldblum
Differential Revision: D6893206
fbshipit-source-id: af47063918a79c851fd39b838d6c63755166e033
Summary: This should make it less flaky
Reviewed By: simpkins
Differential Revision: D6908454
fbshipit-source-id: 07622b469ec635972290c3a4e5d1ed4b05ddd51a
Summary:
Update hg_import_helper.py to parse the repository's .hg/hgrc file, but then
create the repository object with a fresh UI object that has not parsed that
config yet.
This more closely mimic's the behavior of mercurial's dispatch code invoked
when starting the `hg` command line. This behavior is required to ensure that
secondary repository objects that get created end up with the correct
configuration, and do not have settings from the original repository's hgrc
file.
The mercurial behavior of parsing the original repository's hgrc file twice
dates back to rHG741f64dfc04d1.
Reviewed By: quark-zju
Differential Revision: D6909449
fbshipit-source-id: 85073ab6ade4ab70247d48bc670c9924e9e6841f
Summary:
This was causing flakiness in the form of premature timeouts
generating failure status. Looking at the docs for hypothesis, this
option is deprecated and slated to be replaced with an infinite timeout
in the future:
http://hypothesis.readthedocs.io/en/latest/settings.html#hypothesis.settings.timeout
Those docs suggest using `hypothesis.unlimited` instead, but I couldn't import
that symbol (maybe our version is too old?) so I'm just using 0 which means
that no timeout will be applied.
Reviewed By: simpkins
Differential Revision: D6908200
fbshipit-source-id: 1143d4745fc84d9c14b29ff5a2906a4e776d14fe
Summary:
Update `TreeInode::diff()` to check if its hash matches the source control tree
it is being compared to, and return early if they are identical.
I'm surprised that I forgot to include this initially when implementing
`TreeInode::diff()`
This makes `hg status` faster when a large number of unmodified directories
have been loaded.
Reviewed By: chadaustin
Differential Revision: D6890615
fbshipit-source-id: 561630d0220b4875dbf3678161cdb41a8aa4fc82
Summary:
This re-orders some of the code in `TreeInode::diff()` slightly. This should
not affect the behavior of the code.
This moves the `isIgnored` check inside the main `contents_.wlock()` block.
This reduces the number of places where we grab the lock, and will help keep
things simple for an upcoming diff where I need to add some more checks in this
code with the lock held.
This also changes `inodeFuture` to use the new `Future::makeEmpty()`
constructor rather than having to use an `Optional<Future>`
Reviewed By: chadaustin
Differential Revision: D6890616
fbshipit-source-id: 354bbf6a6be6d356fd23e6c0fb6b534679bbe0bb
Summary:
The open source build of Eden with buck has been broken for a while, since buck
removed support for the `thrift_library()` rule.
Lets just remove these files for now since they no longer work. In the short
term it potentially makes sense for us to just use CMake for open source builds
rather than buck.
Reviewed By: chadaustin
Differential Revision: D6893232
fbshipit-source-id: f6e121f1376321b62be42bae60844dfa07fadf65
Summary: We don't think this is happening, but let's test it!
Reviewed By: simpkins
Differential Revision: D6888038
fbshipit-source-id: 754b2ec8f78ff513fd350a74505915e2a1e9ba3e
Summary: Small things I've needed in later diffs.
Reviewed By: wez
Differential Revision: D6877755
fbshipit-source-id: c9002eb0b92dbd8fe9c4f636d2ca79b25cde331f
Summary:
While working on timestamp storage, the fact that
InodeTimestamps was a member of InodeBase kept getting in the way.
Make it its own type.
Reviewed By: simpkins
Differential Revision: D6862835
fbshipit-source-id: 91d8984764f0586b9fa52e961eb5606a530e0416
Summary: This API seems like it should be const, as it does not modify the clock.
Reviewed By: chadaustin, zhupanov
Differential Revision: D6869719
fbshipit-source-id: c8bf4ccab34538b59e6baeedd0b0ff88b328236e
Summary:
A little bit of plumbing to enable the watchman<->eden integration tests on sandcastle.
I noticed that we were still referencing the lame thrift par that I removed a little while back,
so let's clean that up.
Reviewed By: simpkins
Differential Revision: D6863221
fbshipit-source-id: 026f3d316d9ac52fe359cedc4e3c88394ec5c7c4
Summary:
This diff moves the mount-time initialization handling
out of the main loop. This rationale for this is:
* We don't (and shouldn't!) need to process FUSE_INIT for takeover
processing, and this structure allows us to make stronger assertions
about our state.
* we can avoid spinning up multiple threads in the (rare!) case that
the FUSE_INIT fails
* It is now a little harder for exceptions during initialization to
escape our notice.
In rearranging this stuff, I found a race condition in the worker thread
shutdown; we could erroneously emit a completion event before all of
the threads had been torn down and this resulted in sporadic integration
test failures hitting the assertion for the number of joined threads
in the destructor.
Reviewed By: simpkins
Differential Revision: D6766330
fbshipit-source-id: 32afb5a7c739c75aebfdb0a8f896eec5f41ad33f
Summary:
If we fail to lstat a mount because the FUSE daemon is
unconnected, go ahead and unmount it.
Reviewed By: simpkins
Differential Revision: D6848115
fbshipit-source-id: df797b15956db24c4ac87dc6d4cd0a7177dd20ef
Summary:
This is a fairly minimal attempt at asking sandcastle to
run our tests as root. This is inspired by D6608746 but is for
the moment pretty coarse; I just want to see how successful this
is before finishing this off nicely, which means splitting our
contbuild config into two pieces along similar lines as D6608746;
one piece would run the tests as usual without root privileges
whilst the other would run just the integration tests with root
privs.
Reviewed By: chadaustin
Differential Revision: D6612669
fbshipit-source-id: 9a730b02424e939dd3509310b6265d50d1102e77
Summary: Rather than EPERM we get EACCESS when running as root
Reviewed By: chadaustin
Differential Revision: D6853303
fbshipit-source-id: c6743c8d036a24255d2d31c560034c972a0253c7
Summary:
in our sandcastle environment, eden is not currently
installed and the tests that look at the version are throwing
an error when the rpm query fails.
Don't do that!
Reviewed By: simpkins
Differential Revision: D6853305
fbshipit-source-id: d17cb1fb0253ef5ae5000695b6ec5bcda4a6a448
Summary:
This is the spiritual successor to D3302706 which originally
wanted to solve this by adding a python extension. That would prove
to be too painful for the opensource build so it was shelved.
We now need to be able to run our tests in an environment that doesn't
have the `attr` rpm installed so this is a good time to fix this
in a more portable way.
This diff adds a little wrapper around the functions that we already
have for consuming extended attribute information and augments them
with another to list attributes.
The utility emits output in json format and is intended to be fed
directly into the helper functions we have in `fs.py`.
Reviewed By: chadaustin
Differential Revision: D6851182
fbshipit-source-id: 3d1d1a351f2e01405645d45658d1c8bc61a659a4
Summary:
Dir's contents were represented as a vector of 64-bit
pointers to 48-byte structs. This change removes that layer of
indirection, reducing memory usage and slightly pessimizing insertion.
The diff is mostly mechanical outside of the TreeInode.h changes and
calls to emplace..
I'll run memory tests tomorrow, though it's a gamble as to whether
private bytes will show a difference. I may need to shrink the Entry
struct too.
Reviewed By: wez
Differential Revision: D6804957
fbshipit-source-id: b126656dbc7951565e74b6401adde6353e809056
Summary:
This adds version 3 nee 2 of the takeover serialization,
which uses thrift to represent the data. Even though it is logically
version 2 I'm naming it version 3 because we're taking advantage of
the MessageType values from Version 1 which allowed numerical values
1 or 2 to represent different data types.
In this diff we now formalize that first word as the protocol version
and to avoid ambiguity are starting the new version value at 3. I
did briefly consider using the name Version2 to refer to this and setting
the value in the enum to 3, but I didn't want to become known for
API hate crimes against my fellow engineers.
Reviewed By: simpkins
Differential Revision: D6733406
fbshipit-source-id: e2067365e4e8b388490440fd73ab504544011846
Summary:
Whilst chatting with simpkins we realized that we lost
the handshake portion of the takeover protocol during a refactor.
The handshake is important for a couple of reasons:
1. It prevents unmounting and loosing all the mounts in the case
that sometime decides to netcat or otherwise connect to the
socket
2. It gives us an opportunity to short circuit any heavy lifting
if we know that it will be impossible to succeed.
3. It allows us to rollback to earlier builds with older versions
of the takeover protocol.
This diff adds a little bit of machinery to enable passing a set of supported
takeover protocol version numbers. The intent is to retain support for
the two of these at a time; any time we change the encoding/protocol
for takeover we'll bump the version number and add supporting code
to handle the new format, retaining support for the prior version.
Retaining the ability to handle the prior version allows us to downgrade
to an earlier build gracefully if/when the need arises.
I opted to do this here rather than by bumping the `kProtocolID`
constant in `UnixSocket.h` becase we're not really changing the
lowest level of the protocol; just the takeover specific portions.
I haven't actually changed the takeover serialization in this diff,
but do have some work on that happening in D6733406; that diff will
be amended to take advantage and demonstrate how this versioning
scheme works.
A key thing to note about the implementation of this diff is that
the client sends the version number to the server, but doesn't
add any explicit version encoding in the response we receive.
This is deliberate and allows us to upgrade prior builds to
this new scheme. I'll add a more definitive check for this
situation when I actually rev the format in the following diff.
Reviewed By: simpkins
Differential Revision: D6743065
fbshipit-source-id: c991cebfee918daad098105ca6bcfef76374c0ff
Summary:
Tiny thing I noticed when reading code. Keep the entry name
as a StringPiece rather than bouncing through char*.
Reviewed By: zhupanov
Differential Revision: D6820080
fbshipit-source-id: 884e55f74094f44012efbe44b86d8e5903300967
Summary:
isSameAs calls getSha1 which was failing on symlinks. The
original concern was that asking for the SHA-1 of a symlink is
ambiguous: do you want the hash of the symlink or the target? But we
already check for whether you are requesting the SHA-1 of a symlink in
EdenServiceHandler, so it's redundant and incorrect to check in
FileInode too.
Reviewed By: simpkins
Differential Revision: D6847489
fbshipit-source-id: 13966da06bcde75c5c568e09fef14e735de47cfb
Summary: A minor convenience for mass mounting and unmounting.
Reviewed By: simpkins
Differential Revision: D6803003
fbshipit-source-id: 16c0d6982ba0ce2dba9900ee15013fcec8bc5ad5
Summary:
This adds support for the following flags for `hg status` in the Rust wrapper:
* `--all`
* `--modified`
* `--added`
* `--removed`
* `--deleted`
* `--clean`
* `--unknown`
* `--ignored`
* `--no-status`
* `--copies`
There is now a `create_print_config()` function that takes the parsed arguments
and distills them into a `PrintConfig`, which describes the net configuration
as a result of all the flags. (Note that some flags can interfere with each other,
e.g., `--no-status` forces `--copies` to be disabled.)
While here, I cleaned up a bunch of duplication in the code that prints the
status to stdout. Much of the duplication in the old implementation has been
encapsulated in a closure within the new `print_status()` function.
This also made it easier to implement `--copies` for all of the different status
types.
Incidentally, this helps us specify `listIgnored` correctly when calling the
`getScmStatus()` Thrift method as we now keep track of whether the user
specified `--ignored`. This is particularly important for the Eden integration
test suite because its `assert_status()` helper method calls
`hg status --print0 -mardui` by default, so `buck test eden/...` makes much
greater use of the Rust wrapper than it did before.
Reviewed By: wez
Differential Revision: D6770916
fbshipit-source-id: fea1a159addca6d16caa1b7019dc11dc2dcb3e1b
Summary:
In my benchmarks, using fbstring over std::string for PathMap is
worthwhile. There are a lot of entries stored in PathMap and the fact
that fbstring is a word smaller than std::string adds up.
Differential Revision: D6820204
fbshipit-source-id: 1ac34a81828891eb8125b13a074de6ed9715edad
