Summary:
1. Moved the elapsed time calculation and logging for getScmStatus() request to the future completion.
2. Added a helper class "ThriftLogHelper" to attach the time logging to the future completion.
Reviewed By: simpkins
Differential Revision: D7118876
fbshipit-source-id: 5f8212296cd8725a3556c7f5c364ddfa66dbdc95
Summary:
I'm seeing test failures that I have not yet understood and I
thought they might be caused by an implicit conversion from
fusell::InodeNumber to bool. Well, they're not, but this is how I
discovered that. I'm not sure I want to land this change, but I'm
going to leave it around until I figure out what's happening with my
other diffs.
Reviewed By: simpkins
Differential Revision: D7077635
fbshipit-source-id: 50bf67026d2d0da0220c4709e3db24d841960f4b
Summary:
This removes the TARGETS files from the eden github repository. The
open source buck build has been failing for several months, since buck
removed support for the thrift_library() rule.
I will potentially take a stab at adding CMake build support for Eden
at some point in the future.
Reviewed By: chadaustin
Differential Revision: D6893233
fbshipit-source-id: e6023094a807cf481ac49998c6f21b213be6c288
Summary:
mode_t isn't really part of a TreeEntry and I also wanted to see all
the places where we convert an entry type from source control into
mode bits.
Reviewed By: simpkins
Differential Revision: D6983198
fbshipit-source-id: ce1d0976f5fc5130c34a8c93c07a4e26a7cdaf71
Summary:
This is the type of a tree entry, which may be another tree, so
FileType is not an accurate name.
Reviewed By: simpkins
Differential Revision: D6981168
fbshipit-source-id: 997eb8a27f599310ed678ce221c8083722db8bff
Summary:
Adds a SQLite storage implementation and makes it the
default engine for integration tests; this requires fewer resources
to run and the integration tests thus run faster and more reliably.
In the future we may add a configuration option to remember the
storage engine that was used as it is currently not "safe" to switch
from one to the other because the hgproxyhash data cannot be
recreated without re-importing a revision.
Reviewed By: simpkins
Differential Revision: D6919456
fbshipit-source-id: 3afbfafb190cca0e3c797cd9b7cd051768575a8c
Summary:
Rename the existing `state_` variable to `runState_` to help distinguish it
from the new `serverState_` variable.
The information in `runState_` is all related to whether the EdenServer object
is starting, shutting down, or running normally.
Reviewed By: wez
Differential Revision: D6929864
fbshipit-source-id: ad7af381a8a291b12db9308668c7616ebd9b7f39
Summary:
Move all of the privhelper functionality into a PrivHelper class. The
ServerState object now stores the PrivHelper object to use, rather than having
a global singleton.
This will make it easier to stub out the PrivHelper functionality during unit
tests.
Reviewed By: wez
Differential Revision: D6929862
fbshipit-source-id: e3edcb0a03ba9afdf34554cb961fd74557cdd6e3
Summary:
Drop the --thrift_address flag, and always create the thrift socket using a
fixed name under the `.eden` directory. The location of the `.eden` directory
is still configurable with the `--edenDir` argument.
There isn't really much benefit to making the socket path be configurable
separately from the .eden directory path, and it adds to the code complexity.
For instance, while you can tell eden to listen on a TCP socket instead of a
Unix domain socket, that functionality has been broken since D4637285
introduced a `CHECK()` statement that crashes the code when using a TCP socket.
Reviewed By: wez
Differential Revision: D6929863
fbshipit-source-id: ee5f7341d01d3ce522cae936ef3c133bba3f18f7
Summary:
Add a new ServerState class to store process-wide state that is shared across
multiple mounts. Up until now we have passed around the shared state data as
separate variables.
This is intentionally separate from the existing EdenServer class to allow unit
tests to create EdenMount objects without an EdenServer object.
Reviewed By: wez
Differential Revision: D6929861
fbshipit-source-id: 5f22efb6d79dfd70031be1dc37f494c2ad8af902
Summary:
This enables dropping in alternative implementations
of LocalStore and adds a MemoryLocalStore implementation for
use in our tests.
This diff doesn't change the default storage option for the
eden server. I'll look at adding such an option in a follow up diff.
Reviewed By: chadaustin
Differential Revision: D6910413
fbshipit-source-id: 018bf04e0bff101e1f0ab35e8580ca2a2622e5ef
Summary:
Change `LoggerDB::get()` to a reference instead of a pointer since this
function can never return null.
Reviewed By: yfeldblum
Differential Revision: D6893206
fbshipit-source-id: af47063918a79c851fd39b838d6c63755166e033
Summary:
Dir's contents were represented as a vector of 64-bit
pointers to 48-byte structs. This change removes that layer of
indirection, reducing memory usage and slightly pessimizing insertion.
The diff is mostly mechanical outside of the TreeInode.h changes and
calls to emplace..
I'll run memory tests tomorrow, though it's a gamble as to whether
private bytes will show a difference. I may need to shrink the Entry
struct too.
Reviewed By: wez
Differential Revision: D6804957
fbshipit-source-id: b126656dbc7951565e74b6401adde6353e809056
Summary:
It's interesting to see the total number of loaded files
vs. trees when the loaded inode count is high.
Reviewed By: wez
Differential Revision: D6765874
fbshipit-source-id: 178b30184428bd5cf5e005eb475e4f5a1476c385
Summary:
I'm not sure what was wrong with the old code, but I
simplified and clarified all of the time math and now `eden debug
unload` behaves as I'd expect it should.
Reviewed By: simpkins
Differential Revision: D6764962
fbshipit-source-id: 3ed359d4ab4652e95d1538a0982c24185999351c
Summary:
Add a log message on successful shutdown, just so that Eden's shutdown point is
easily identifiable in the log files.
Reviewed By: wez
Differential Revision: D6757386
fbshipit-source-id: f19a37c2ec9ae64fff99d40201d80e5f8faeaea5
Summary:
akushner mentioned that he saw a hang when performing
a graceful restart and we realized that his cwd was inside the
mount point that he was taking over. While this doesn't guarantee
problems (I've run it this way almost exclusively with no issues)
it feels like a potential problem to hold a reference to a file
descriptor within a fuse mount that we are ourselves going to
manage: it seems likely that we could end up deadlocking if
we were to attempt any IO on such a descriptor.
This diff seeks to head off this particular case by simply
chdiring to `/` very early on startup. This is typically
something that is handled by the `daemonize` function in
old school unix code.
Reviewed By: simpkins
Differential Revision: D6740547
fbshipit-source-id: 986004e9ed6cfa493ad55c5cdbca1805fc0f7768
Summary:
This avoids the long delay and the `Cpp2Worker.cpp:281] Failed to join
outstanding requests.` message from showing up in the eden logs during shutdown
and graceful restart.
Reviewed By: chadaustin
Differential Revision: D6693117
fbshipit-source-id: 3c56314f288a596264cddae0d8bbab66ab19e9fe
Summary:
Previously, `EdenMount::create` would implicitly call
`EdenMount::initialize` which would load the root inode and the `.eden` magical
directory. That's fine for the fresh mount case, but in the case of the
graceful restart we need to take the opportunity to apply the `InodeMap`
from the old process before we start muddying its state.
This diff breaks out the `initialize` method from the `create` method and
makes the mount code call it after potentially loading the `InodeMap` from
the takeover data.
In addition, this diff removes the the `root->loadMaterializedChildren()`
call from the mount initialization code. It is no longer required to do
this eagerly and it makes things simpler and our memory profile a little
smaller to defer this (I haven't measured how much that impacts things).
Reviewed By: simpkins
Differential Revision: D6691182
fbshipit-source-id: 52033a6d64105b658314a919f69dbfcd4eea242b
Summary:
This removes the duplicated logic and makes it a little
bit easier to follow the initialization sequence.
It doesn't change the behavior, just moves some code around.
Reviewed By: simpkins
Differential Revision: D6691180
fbshipit-source-id: 2068dbe56ebe9a6136d69689997aec8dedd32be0
Summary:
This is the "simple" threading through of a `doTakeover`
flag and the return of `Optional<TakeoverData>` in the unmount code.
Reviewed By: simpkins
Differential Revision: D6691181
fbshipit-source-id: 4a384787783c16085f2e9964964023ba07cefca3
Summary:
I'm so-so on a bit of the implementation here, but it works!
I had to change the `takeoverPromise` from the `pair<fuseDevice, connInfo>`
to a new helper struct because we now have three distinct pieces of data
to pass out of EdenMount to build up the overall TakeoverData.
The key change in this diff is that we have to release all of the file handles
we're maintaining in the `FileHandleMap` prior to shutting down the `InodeMap`,
otherwise the `InodeMap` will never complete (it waits for all inodes to be
unreferenced, and that cannot happen while there are open file handles). I've
made the `FileHandleMap` serialization and clearing contingent on performing a
takeover shutdown because that feels like the safest thing to do wrt. not
losing any pending writes.
Reviewed By: simpkins
Differential Revision: D6672437
fbshipit-source-id: 7b1f0f8e7ff09dbed850c7737383ecdf1e5ff0c7
Summary:
This is the key portion that makes the graceful restart
function. This diff connects almost all of the moving pieces together;
it informs the priv helper about the takeover mount and transfers
the InodeMap information into the new generation of the eden server.
It doesn't yet load the fileHandleMap (will tackle that in a follow up diff)
Reviewed By: simpkins
Differential Revision: D6670903
fbshipit-source-id: 1770d99eb1477440a6c1deed83b0da55b9c1bbe4
Summary:
this isn't how we really want to do this long term, it's
just the most expedient short term implementation.
This diff provides an implementation of the `InodeMap::save()` which
was previously a stub method; the new implementation returns a thrift
structure that encompasses the unloaded inodes in the map, and adds
a corresponding load() method that performs the reverse transformation.
The struct is serialized into the Takeover data.
This diff doesn't hook up the real serialized data to EdenServer; that will happen
in a follow-on diff.
The way that we actually want to handle this longer term is to store the
`numFuseReferences` field into the overlay file on disk, but to do so we
will need to add a mountGeneration field alongside it and ensure that we
always write out the correct information at the correct times. In addition,
we'd need to add equivalent data to TreeInode::Entry and add accessors that
safely return the correct values in the correct situations.
In the interest of getting something working, I've just dumped this code in
here.
I've also placed the thrift structure for this in `fuse/handlemap.thrift`;
this is a slight layering violation but one that feels "OK" in light of
the imminent refactor of the Takeover data struct to be its own thrift
struct anyway.
Reviewed By: simpkins
Differential Revision: D6670904
fbshipit-source-id: 11a0918954c741935c587e46fcb0e38849010de1
Summary:
This puts the data into the takeover information during takeover
shutdown, but doesn't do anything to pull it out again (that will be in a follow on diff).
The serialization stuff could be done a little bit more efficiently (since we
will perform an extra thrift serialization step just to compute the length, and
repeat it again later), but we're planning on replacing this with thrift
serializing soon, once simpkins diff stack lands, so I'm not losing sleep over
it.
Reviewed By: simpkins
Differential Revision: D6668846
fbshipit-source-id: e6d01428bd506a9e93b427db499770fce0a0983a
Summary:
This fulfils a TODO but doesn't do anything useful
with that data at this time.
Reviewed By: simpkins
Differential Revision: D6552750
fbshipit-source-id: 0c441fd0c2ab43785b4d98c4ca6ff643a20629e0
Summary:
This adds some plumbing to thread the fuse device descriptor and
negotiated capabilities through to the takeover code.
I initially wanted to just make the
unmount future yield the device descriptor, but since that uses
`SharedPromise` it is not compatible with a move-only type like
`folly:File`, so I added an optional promise to deal with just that.
I'm also populating the takeover mount information (path, bind mounts)
for each mount point.
Reviewed By: simpkins
Differential Revision: D6494509
fbshipit-source-id: a90684292dc1d8e06ce2c0721eadd8d393377f33
Summary:
This serves a few purposes:
1. We can avoid some conditional code inside eden if we know that
we have a specific fuse_kernel.h header implementation.
2. We don't have to figure out a way to propagate the kernel
capabilities through the graceful restart process.
3. libfuse3 removed the channel/session hooks that we've been
using thus far to interject ourselves for mounting and
graceful restarting, so we were already effectively the
walking dead here.
4. We're now able to take advtange of the latest aspects of
the fuse kernel interface without being tied to the implementation
of libfuse2 or libfuse3. We're interested in the readdirplus
functionality and will look at enabling that in a future diff.
This may make some things slightly harder for the more immediate
macOS port but I belive that we're in a much better place overall.
This diff is relatively mechanical and sadly is (unavoidably) large.
The main aspects of this diff are:
1. The `fuse_ino_t` type was provided by libfuse so we needed to
replace it with our own definition. This has decent penetration
throughout the codebase.
2. The confusing `fuse_file_info` type that was multi-purpose and
had fields that were sometimes *in* parameters and sometimes *out*
parameters has been removed and replaced with a simpler *flags*
parameter that corresponds to the `open(2)` flags parameter.
The *out* portions are subsumed by existing file handle metadata
methods.
3. The fuse parameters returned from variations of the `LOOKUP` opcode
now return the fuse kernel type for this directly. I suspect
that we may need to introduce a compatibility type when we revisit
the macOS port, but this at least makes this diff slightly simpler.
You'll notice that some field and symbol name prefixes vary as
a result of this.
4. Similarly for `setattr`, libfuse separated the kernel data into
two parameters that were a little awkward to use; we're now just
passing the kernel data through and this, IMO, makes the interface
slightly more understandable.
5. The bulk of the code from `Dispatcher.cpp` that shimmed the
libfuse callbacks into the C++ virtual methods has been removed
and replaced by a `switch` statement based dispatcher in
`FuseChannel`. I'm not married to this being `switch` based
and may revise this to be driven by an `unordered_map` of
opcode -> dispatcher method defined in `FuseChannel`. Regardless,
`Dispatcher.cpp` is now much slimmer and should be easier to
replace by rolling it together into `EdenDispatcher.cpp` in
the future should we desire to do so.
6. This diff disables dispatching `poll` and `ioctl` calls. We
didn't make use of them and their interfaces are a bit fiddly.
7. `INTERRUPT` is also disabled here. I will re-enable it in
a follow-up diff where I can also revise how we track outstanding
requests for graceful shutdown.
8. I've imported `fuse_kernel.h` from libfuse. This is included
under the permissive 2-clause BSD license that it allows for
exactly this integration purpose.
Reviewed By: simpkins
Differential Revision: D6576472
fbshipit-source-id: 7cb088af5e06fe27bf22a1bed295c18c17d8006c
Summary:
This is a codemod to change from using @/ to // in basic cases.
- TARGETS files with lines starting with @/ (but excluding @/third-party:
- autodeps lines in source and TARGETS files ( (dep|manual)=@/ ), excluding @/third-party
- Targets in string macros
The only thing left of the old format should be @/third-party:foo:bar
drop-conflicts
Reviewed By: ttsugriy
Differential Revision: D6605465
fbshipit-source-id: ae50de2e1edb3f97c0b839d4021f38d77b7ab64c
Summary:
`eden stats` used to show system memory usage which was not very
interesting (and can be gleaned from top). Instead read the contents
of /proc/self/smaps and sum the Private_Dirty fields to get a number
that more accurately reflects impact on the rest of the system.
Reviewed By: wez
Differential Revision: D6575595
fbshipit-source-id: 9badc5cd5a1b56d3ccb27edd1a2d20ee74ec34ae
Summary:
During edenfs startup, use `realpath()` or `normalizeBestEffort()` to resolve
symlinks in the input configuration paths if possible.
Reviewed By: chadaustin
Differential Revision: D6527494
fbshipit-source-id: 4377099e8c65217fd128c06de77d50f4316f4fc7
Summary:
The RocksDB location is relative to the .eden directory. Given that the
location of the .eden directory can be controlled from the command line it
doesn't seem worthwhile to make the RocksDB location independently controllable
too.
Reviewed By: chadaustin
Differential Revision: D6527495
fbshipit-source-id: dab8c22f3f1a74de908ea33d0b20c4115c28ce31
Summary:
Add EdenCPUThreadPool and UnboundedQueueThreadPool types to make it clearer
that it's always okay for prefetch, deferred diff entry, and hg import to
shuttle work back to the main thread pool.
This diff changes no behavior - it just makes some invariants explicit.
Reviewed By: wez, simpkins
Differential Revision: D6504117
fbshipit-source-id: 3400ad55c00b3719ecba31807fd992442f622cd9
Summary: Ensure everything remaining in dependencies of `folly:folly` that has < 500 references is explicit referenced.
Reviewed By: yfeldblum
Differential Revision: D6540137
fbshipit-source-id: 0a2ae5cf775278eedcccdb914688890acd12dab7
Summary: And then, there was 1, left all alone, preventing the whole thing from collapsing in on itself.
Reviewed By: yfeldblum
Differential Revision: D6469584
fbshipit-source-id: 4ea1fbf97ad466bc34f2e682394d328c97e539ba
Summary:
Use an unbounded queue for edenfs's main thread pool. This fixes a
crash where DeferredDiffEntry multigets a batch of trees and pushes
the completion callbacks back onto the server thread pool. If the
server thread pool is bounded and throws when the queue is full, then
the import fails.
There is a slight performance hit relative to LifoSemMPMCQueue but
hopefully it isn't a big deal. An unbounded lock-free queue would be
nicer.
Reviewed By: simpkins
Differential Revision: D6490979
fbshipit-source-id: bc55dd6526f0ceb9d8b5e43a1a275250a9838aca
Summary:
Introduce a Clock seam. This will allow us to write tests around
ctime, mtime, and atime logic.
Reviewed By: wez
Differential Revision: D6392543
fbshipit-source-id: 1721d76d2364b135b4ef5c078ef60f7f8526259e
Summary:
Replace the initLoggingGlogStyle() function with a more generic initLogging()
function that accepts a log config string to be parsed with parseLogConfig().
Reviewed By: bolinfest, yfeldblum
Differential Revision: D6342086
fbshipit-source-id: fb1bffd11f190b70e03e2ccbf2b30be08d655242
Summary:
Previously, we used the Mercurial code `g` when faced with an `UNTRACKED_ADDED`
file conflict, but that was allowing merges to silently succeed that should not
have. This revision changes our logic to use the code `m` for merge, which
unearthed that we were not honoring the user's `update.check` setting properly.
Because we use `update.check=noconflict` internally at Facebook, we changed the
Eden integration tests to default to verifying Hg running with this setting. To
support it properly, we had to port this code from `update.py` in Mercurial to
our own `_determine_actions_for_conflicts()` function:
```
if updatecheck == 'noconflict':
for f, (m, args, msg) in actionbyfile.iteritems():
if m not in ('g', 'k', 'e', 'r', 'pr'):
msg = _("conflicting changes")
hint = _("commit or update --clean to discard changes")
raise error.Abort(msg, hint=hint)
```
However, this introduced an interesting issue where the `checkOutRevision()`
Thrift call from Hg would update the `SNAPSHOT` file on the server, but
`.hg/dirstate` would not get updated with the new parents until the update
completed on the client. With the new call to `raise error.Abort` on the client,
we could get in a state where the `SNAPSHOT` file had the hash of the commit
assuming the update succeeded, but `.hg/dirstate` reflected the reality where it
failed.
To that end, we changed `checkOutRevision()` to take a new parameter,
`checkoutMode`, which can take on one of three values: `NORMAL`, `DRY_RUN`, and
`FORCE`. Now if the user tries to do an ordinary `hg update` with
`update.check=noconflict`, we first do a `DRY_RUN` and examine the potential
conflicts. Only if the conflicts should not block the update do we proceed with
a call to `checkOutRevision()` in `NORMAL` mode.
To make this work, we had to make a number of changes to `CheckoutAction`,
`CheckoutContext`, `EdenMount`, and `TreeInode` to keep track of the
`checkoutMode` and ensure that no changes are made to the working copy when a
`DRY_RUN` is in effect.
One minor issue (for which there is a `TODO`) is that a `DRY_RUN` will not
report any `DIRECTORY_NOT_EMPTY` conflicts that may exist. As `TreeInode` is
implemented today, it is a bit messy to report this type of conflict without
modifying the working copy along the way.
Finally, any `UNTRACKED_ADDED` conflict should cause an update to
abort to match the behavior in stock Mercurial if the user has the following
config setting:
```
[commands]
update.check = noconflict
```
Though the original name for this setting was:
```
[experimental]
updatecheck = noconflict
```
Although I am on Mercurial 4.4.1, the `update.check` setting does not seem to
take effect when I run the integration tests, but the `updatecheck` setting
does, so for now, I set both in `hg_extension_test_base.py` with a `TODO` to
remove `updatecheck` once I can get `update.check` to do its job.
Reviewed By: simpkins
Differential Revision: D6366007
fbshipit-source-id: bb3ecb1270e77d59d7d9e7baa36ada61971bbc49
Summary:
The goal is to provide a fast path for watchman to flesh
out the total set of changed files when it needs relay that information
on to consumers.
We choose not to include the full list in the Journal when checking out
between revisions because it will not always be needed and may be an
expensive `O(repo)` operation to compute. This means that watchman
needs to expand that information for itself, and that is currently
a fairly slow query to invoke through mercurial.
Since watchman is responding to journal events from eden we know that
we have tree data for the old and new hashes and thus we should be
able to efficiently compute that diff.
This implementation is slightly awful because it will instantiate an
unlinked TreeInode object for one side of the query, and will in
turn populate any children that differ as it walks down the tree.
A follow on diff will look at making a flavor of the diff code that
can diff raw Tree objects instead.
Reviewed By: bolinfest
Differential Revision: D6305844
fbshipit-source-id: 7506c9ba1f4febebcdc283c414261810a3951588
Summary: It is currently unused. Let's bring it back if/when we need it.
Reviewed By: chadaustin
Differential Revision: D6368867
fbshipit-source-id: 096015ba597a6e04f544273ba9773576429e39ce
Summary:
Define constants at the top of EdenServer.cpp for the names of the main lock
file, the thrift socket, and the takeover socket.
Reviewed By: bolinfest
Differential Revision: D6295040
fbshipit-source-id: 8605840a50c84bc89b798123d1063bbb11ff2502
Summary:
This begins implementing the "client-side" portion of graceful takeover in
edenfs. When the --takeover flag is specified, if edenfs finds that another
edenfs process is already running it will attempt to gracefully take over its
mount points rather than exiting with an error.
This does not yet actually take over the mount points themselves--it still
sends dummy mount information during shutdown, and does not use this data
during startup. However, we do perform takeover of the eden lock file and the
thrift server socket.
Reviewed By: bolinfest
Differential Revision: D6038944
fbshipit-source-id: 42406a0559367cec79af088b4ca84c22de3f3ef3
Summary:
Update the TakeoverData to also include the thrift server socket.
Also update EdenServer to set this field when performing a takeover
shutdown.
Reviewed By: bolinfest
Differential Revision: D6038945
fbshipit-source-id: 725faa431b3b55d617ef645c8a7eae080e4fe066
Summary:
Update EdenServer to implement the TakeoverHandler API, and to exit after
sending the mount point takeover data. The actual shutdown logic itself is not
implemented yet--this just sends dummy data for now. However, this does serve
as a proof of concept that the TakeoverServer and TakeoverClient code functions
as desired.
Reviewed By: bolinfest
Differential Revision: D6018180
fbshipit-source-id: c19581928926a46b767f1ee5c1761381e5055fa9
Summary:
The headline changes of this revision are:
- Changes the format of the config file from INI to TOML
(the `edenrc` file under `~/local/.eden` has been replaced
with `config.toml`). This revision includes logic for automatically
performing the migration when Eden is restarted.
- Inlines data from `/etc/eden/config.d` into the TOML file.
Historically, the `edenrc` file for a client would contain the
name of the "configuration alias" defined in a config file like
`~/.edenrc` or `/etc/eden/config.d/00-defaults`. When Eden
loaded a client, it would have to first read the `edenrc` and
then reconstitute the rest of the client configuration by
looking up the alias in the set of config files that were used to
create the client in the first place.
This changes things so that all of the data that was being
cross-referenced is now inlined in the client's config file.
This makes loading a config considerably simpler at the cost
of no longer being able to change the config for multiple clients
that were cloned from the same configuration alias in one place.
It was questionable whether being able to modify a client from
a foreign config after it was created was a safe thing to do, anyway.
Eliminating the need for a historic link to the configuration alias
will make it easier to support running `eden clone` on an arbitrary
local Hg or Git repo. So long as `eden clone` can extract enough
information from the local repo to create an appropriate config file
for the new Eden client, there is no need for a configuration alias
to exist a priori.
Since we were already changing the data in the config file, this
seemed like an appropriate time to make the switch from INI to
TOML, as this was something we wanted to do, anyway.
In testing, I discovered a discrepancy between how boost's
`boost::property_tree::ptree` and Python's `ConfigParser` handled
the following section heading:
```
[repository ZtmpZsillyZeden-clone.LIkh32]
```
Apparently `hasSection("repository ZtmpZsillyZeden-clone.LIkh32")`
in boost would fail to find this section. Because
[[https://stackoverflow.com/questions/13109506/are-hyphens-allowed-in-section-definitions-in-ini-files | there is no spec for INI]],
it is not that surprising that boost and `ConfigParser` do not 100% agree
on what they accept. Moving to TOML means we have a configuration
language with the following desirable properties:
- It has a formal spec, unlike INI. This is important because there are parsers
in a wide range of programming languages that, in theory, accept a consistent
input language.
- It is reasonable for humans to write, as it supports comments, unlike JSON.
- It supports nested structures, like maps and arrays, without going crazy
on the input language it supports, unlike YAML.
Eden now depends on the following third-party TOML parsers:
* C++ https://github.com/skystrife/cpptoml
* Python https://github.com/uiri/toml
This revision also changes the organization of `~/local/.eden` slightly. For now,
there is still a `config.json` file, but the values are no longer hashes of the realpath
of the mount. Instead, we take the basename of the realpath and use that as the
name of the directory under `~/local/.eden/clients`. If there is a naming collision, we
add the first available integral suffix. Using the basename makes it easier to
navigate the `~/local/.eden/clients` directory.
Although the `edenrc` file under `~/local/.eden/clients` has been switched from INI
to TOML, the other Eden config files (`~/.edenrc` and `/etc/eden/config.d/*`) still use
INI. Migrating those to TOML will be done in a future revision.
Note this revision allowed us to eliminate `facebook::eden::InterpolatedPropertyTree`
as well as a number of uses of boost due to the elimination of
`ClientConfig::loadConfigData()` in the C++ code. Because `ClientConfig`
no longer does interpolation, a bit of `ClientConfigTest` was deleted as part of
this revision because it is no longer relevant.
Reviewed By: wez
Differential Revision: D6310325
fbshipit-source-id: 2548149c064cdf8e78a3b3ce6fe667ff70f94f84
Summary:
Have HgBackingStore hold multiple HgImporters in its own thread pool. Incoming
requests are processed by threads in the thread pool.
Reviewed By: wez
Differential Revision: D6265043
fbshipit-source-id: b2d4f345b772f296c5335a7fbcadfce1d93245fd
Summary:
Moving the post-clone step out of C++ makes it so that
`ClientConfig` in C++ no longer knows about `hooks` and requires
`RepoConfig` to know about `hooks`. This helps us reduce
`ClientConfig`'s dependency on `ConfigData`, as my next step
is to move the remaining information that `ClientConfig` gets from
`ConfigData` directly into the client's `edenrc` file under
`~/local/.eden`.
Reviewed By: chadaustin
Differential Revision: D6310544
fbshipit-source-id: dec7a21281ab49e0416b8872757970a4eff2d943
Summary:
This is another, more efficient way to fix the mode/opt build issues
seen in D6272580. When C++17 is released, it could be updated to
std::size.
Reviewed By: yfeldblum
Differential Revision: D6274205
fbshipit-source-id: 07f25a1a0f6575b80b7d1a8af7b7c723f0f9e4f0
Summary:
This is follow-up to the lock ordering issues in
StreamingSubscriber. The Journal locks are now finer-grained and no
locks are held while the subscribers are invoked. This change
prevents future deadlocks.
Reviewed By: wez
Differential Revision: D6281410
fbshipit-source-id: 797c164395831752f61cc15928b6d4ce4dab1b68
Summary:
Tweak the `INSTRUMENT_THRIFT_CALL()` log levels in EdenServiceHandler.
For the most part this changes the code so that modifying calls (e.g.,
`mount`, `checkOutRevision`, etc) are logged at `DBG2` and higher, while
read-only calls (`getSHA1`, `getParentCommits`) are logged at `DBG3` and
lower. Since we log all eden messages at `DBG2` by default this means that
modifying calls will be enabled by default but read-only calls will be
disabled.
Some important read-only calls such as `getScmStatus` and
`getFilesChangedSince` do still log at the `DBG2` level.
The main motivation for this is that `buck build` often triggers many thousands
of separate `getSHA1` calls. It doesn't seem terribly valuable to flood the
eden logs with these messages by default. These can always be enabled at
runtime if desired when debugging buck performance.
Reviewed By: bolinfest
Differential Revision: D6295566
fbshipit-source-id: dd344c1dea773f4f2a56e2b0dbb18b8738303944
Summary: Make this function match our C++ guidelines.
Reviewed By: wez
Differential Revision: D6288591
fbshipit-source-id: 1a4f52a8c1e0497df938533fe29da10264eb1ccf
Summary: We were previously potentially deleting the include prefix. We also weren't using the cpp2 include prefix if the global one wasn't set. This makes sure we use it, and fixes a bug where 'X_types.h' was included without a full prefix.
Reviewed By: yfeldblum
Differential Revision: D6236108
fbshipit-source-id: 076747fcab2b1414bafa42c9e481ba1e1e5df4b1
Summary:
This setting is bad for a few reasons
1) there is no correct value; rejecting connections is typically something you want to do when under pressure, which no fixed number of connections can indicate
2) it's not discoverable when it trips, and pretty much always confuses people
3) the effect is generally not better than the thing it is theoretically supposed to prevent, when it trips, servers crash and exhibit other weird behaviors; I don't think I've ever seen a situation where I thought this was needed, and I've seen many were it created problems where none would have existed.
About a year ago, I removed almost all calls to this; however, I left some behind since they were slightly harder to clean up or had unique flag names. Since then, 2 things have happened
1) more copypasta instances have cropped up
2) More people have run into issues with this flag, notably up2x recently
This removes all calls except one (realtime Pylon has some somewhat more complex config here, I'll need to talk to them). This is somewhat aggressive; some of the calls are totally safe to remove, as they're either copypasted or set it to 0 or some absurdly high number. Others are less obvious. I've decided to adopt a door-in-the-face strategy here and see if anyone tells me I should be more conservative
I couldn't delete all the flags; these ones are in use
zdb_thrift_max_connection
thrift2_max_connections (commerce ranker)
max_connections (a bunch of places)
maxConnections (presence)
I'll need a separate diff/set of diffs to delete those
Reviewed By: eduardo-elizondo, yfeldblum
Differential Revision: D6182563
fbshipit-source-id: e778edd9da582f4ca90a902621cb49f1e04ca26e
Summary:
Building with gcc/opt mode complains that `strlen()` is not a constant
expression, so I'm dropping the `constexpr` to fix the opt build.
Reviewed By: chadaustin
Differential Revision: D6272580
fbshipit-source-id: a8e3a93ea04e878353c6cab31cb0b1b4705276fe
Summary:
In Eden, some Thrift endpoints get a lot of hits whereas others are used less
frequently. By giving each endpoint its own logging category, we can toggle the
logging for each one independently.
Each of our Thrift endpoint methods should start with the following macro:
INSTRUMENT_THRIFT_CALL();
then within the method, the macro `TLOG()` should be used everywhere
`XLOG(LEVEL)` would normally used. `LOG()` ensures the logger with the category
specific to the method will be used.
For an endpoint named `XXX`, the logging category will be `eden.thrift.XXX`.
Reviewed By: simpkins
Differential Revision: D6108311
fbshipit-source-id: 23a34179811359b0819434de31a3601d29c3b4f0
Summary:
Opening the RocksDB store can take a long time if there is a fair amount of
data in the store. In my current .eden directory I have around 10GB of data in
the local store, and it takes RocksDB nearly 60 seconds to open the database.
These log messages help provide a little more information about what edenfs is
doing during this startup delay.
Reviewed By: bolinfest
Differential Revision: D6263603
fbshipit-source-id: a0945aa1cc020b95944b365b17869756dcc27407
Summary:
This is a major change to how we manage the dirstate in Eden's Hg extension.
Previously, the dirstate information was stored under `$EDEN_CONFIG_DIR`,
which is Eden's private storage. Any time the Mercurial extension wanted to
read or write the dirstate, it had to make a Thrift request to Eden to do so on
its behalf. The upside is that Eden could answer dirstate-related questions
independently of the Python code.
This was sufficiently different than how Mercurial's default dirstate worked
that our subclass, `eden_dirstate`, had to override quite a bit of behavior.
Failing to manage the `.hg/dirstate` file in a way similar to the way Mercurial
does has exposed some "unofficial contracts" that Mercurial has. For example,
tools like Nuclide rely on changes to the `.hg/dirstate` file as a heuristic to
determine when to invalidate its internal caches for Mercurial data.
Today, Mercurial has a well-factored `dirstatemap` abstraction that is primarily
responsible for the transactions with the dirstate's data. With this split, we can
focus on putting most of our customizations in our `eden_dirstate_map` subclass
while our `eden_dirstate` class has to override fewer methods. Because the
data is managed through the `.hg/dirstate` file, transaction logic in Mercurial that
relies on renaming/copying that file will work out-of-the-box. This change
also reduces the number of Thrift calls the Mercurial extension has to make
for operations like `hg status` or `hg add`.
In this revision, we introduce our own binary format for the `.hg/dirstate` file.
The logic to read and write this file is in `eden/py/dirstate.py`. After the first
40 bytes, which are used for the parent hashes, the next four bytes are
reserved for a version number for the file format so we can manage file format
changes going forward.
Admittedly one downside of this change is that it is a breaking change.
Ideally, users should commit all of their local changes in their existing mounts,
shutdown Eden, delete the old mounts, restart Eden, and re-clone.
In the end, this change deletes a number of Mercurial-specific code and Thrift
APIs from Eden. This is a better separation of concerns that makes Eden more
SCM-agnostic. For example, this change removes `Dirstate.cpp` and
`DirstatePersistance.cpp`, replacing them with the much simpler and more
general `Differ.cpp`. The Mercurial-specific logic from `Dirstate.cpp` that turned
a diff into an `hg status` now lives in the Mercurial extension in
`EdenThriftClient.getStatus()`, which is much more appropriate.
Note that this reverts the changes that were recently introduced in D6116105:
we now need to intercept `localrepo.localrepository.dirstate` once again.
Reviewed By: simpkins
Differential Revision: D6179950
fbshipit-source-id: 5b78904909b669c9cc606e2fe1fd118ef6eaab95
Summary:
Per discussion with bolinfest, this brings Eden in line with clang-format.
This diff was generated with `find . \( -iname '*.cpp' -o -iname '*.h' \) -exec bash -c "yes | arc lint {}" \;`
Reviewed By: bolinfest
Differential Revision: D6232695
fbshipit-source-id: d54942bf1c69b5b0dcd4df629f1f2d5538c9e28c
Summary:
D5067763 introduced a potential deadlock. The issue is that all of the FUSE threads were blocked
on the HgImporter thread pool, which completes its futures back on serverEventThread_. The
FUSE threads were blocked on Future::get() in ensureDataLoaded().
Eventually, the right fix is some combination of eliminating ensureDataLoaded() and
replacing it with an explicitly-asynchronous API.
Reviewed By: bolinfest
Differential Revision: D6212858
fbshipit-source-id: 42b17d3e20a200f26b87588784edb5ee51e96a4a
Summary:
The result of this call was never used, and the call does not appear to have
any side-effects.
Reviewed By: chadaustin
Differential Revision: D6212076
fbshipit-source-id: e87858b75d9c0482e5f544411a85ebf1a66d63ba
Summary:
For a while, the recommended approach (e.g. from folks on the thrift team, from folks on the javafoundations team) has been to use `languages = ["java-swift"]`. This renaming brings the code in line with the recommendation. There are no actual changes to any builds, this is just a rename.
This is the second diff in a small stack. It's a fully automated change that uses buildozer to rename `*-java` rules to `*-javadeprecated`.
Differential Revision: D6189950
fbshipit-source-id: a08ba86cb92293a07eceb4b6b29385c7bf647b72
Summary:
For a while, the recommended approach (e.g. from folks on the thrift team, from
folks on the javafoundations team) has been to use `languages = ["java-swift"]`.
This renaming brings the code in line with the recommendation. There are no
actual changes to any builds, this is just a rename.
This is the first diff in a small stack. It edits the `JavaThriftConverter` to
use the `-javadeprecated` naming, but with a `java_library` shim that retains
the `-java` naming temporarily (to ease the transition). There's also a little
shim when processing the `languages` parameter, to accept either `java` or
`javadeprecated`.
Reviewed By: yfeldblum
Differential Revision: D6204449
fbshipit-source-id: 3e7f3ca7a4d5e1976176479ff4298da72af9e685
Summary:
There was a lock ordering violation in the combination of StreamingSubscriber and Journal. I changed StreamingSubscriber to always acquire the Journal lock before StreamingSubscriber's State lock.
I also simplified StreamingSubscriber's API to a single static function to isolate all of the lifetime and thread safety concerns in one implementation file.
This fixes a regression caused by D6162717
Reviewed By: simpkins
Differential Revision: D6202770
fbshipit-source-id: 326269db15bf3200bd6321edf372daf286784fb5
Summary:
We have a couple of issues with the watchman/eden integration:
1. If you "win" the race, you can cause a segfault during shutdown
by changing files during unmount. This causes the journal updating
code to trigger a send to the client, but the associated eventBase
has already been destroyed.
2. We don't proactively send any signal to the subscriber (in practice: watchman)
when we unmount. Watchman detects the eden shutdown by noticing that its
socket has closed but has no way to detect an unmount.
This diff tries to connect the unmount with the set of subscribers and tries
to cause the thrift socket to close out.
Reviewed By: bolinfest
Differential Revision: D6162717
fbshipit-source-id: 42d4a005089cd9cddf204997b1643570488f04c3
Summary:
Previously, the `savebackup()` and `restorebackup()` methods in `eden_dirstate`
only retained the parent commit hashes. With this change, now the dirstate tuples
and entries in the copymap for the dirstate are also included as part of the saved
state.
Failing to restore all of the state caused issues when doing things like aborting
an `hg split`, as observed by one of our users. Although this fix works, we ultimately
plan to move the responsibility for persisting dirstate data out of Eden and into the
Hg extension. Then the data will live in `.hg/dirstate` like it would for the default
dirstate implementation.
Reviewed By: simpkins
Differential Revision: D6145420
fbshipit-source-id: baa077dee73847a47cc171cd980cdd272b3a3a99
Summary: This diff helps some common pitfalls when using set_log_level.
Reviewed By: simpkins
Differential Revision: D6142849
fbshipit-source-id: 7fa35392dda148af90d0aefdb872b6e8a8b770db
Summary:
Thess flags are not used (neither in code, nor passed in on command line). List of flags that are getting killed:
thriftMaxNumMessagesInQueue
max_queue_message_num
thrift_max_unprocessed_message
thrift_set_max_num_messages_in_queue
decorator_thriftSetMaxNumMessagesInQueue
atlas_pinger_imps_thriftSetMaxNumMessagesInQueue
tesseract_ocr_max_queue_msgs_per_thread
max_msgs_in_queue
max_num_messages_in_queue
maxNumMessagesInQueue
thrift_max_messages
thrift_queue_len
max_accepts_in_pipe
max_pending_connections_per_io_thread
thriftMaxNumPendingConnectionsPerWorker
lookup_thriftSetMaxNumMessagesInQueue
thriftSetMaxNumPendingConnectionsPerWorker
max_queue_msgs_per_thread
ds_thrift_max_num_messages_in_queue
zdb_thrift_max_num_messages_in_queue
update_control_proxy_queue_length
safety_enforcer_proxy_queue_length
maxRequestsInQueue
max_num_messages_in_pipe
Reviewed By: yfeldblum
Differential Revision: D6143500
fbshipit-source-id: 541507d50100817590b91cdd48e39a29e7c465ea
Summary:
Have HgBackingStore hold multiple HgImporters in its own thread pool. Incoming
requests are processed by threads in the thread pool.
Reviewed By: wez
Differential Revision: D5067763
fbshipit-source-id: d666b1026455d13442367673010b5934ff4cdb48
Summary:
While debugging SRProxy issue I found a bug in ThriftServer initialization (more details in D6115379), due to that bug we always have accept queue of 1k and the value passed to `setMaxNumPendingConnectionsPerWorker` is ignored, thus making all call sites bogus.
We have ~350 calls to this function in fbcode, all of them pass random numbers such as 50, 100, 50k, etc. I’m removing all of them before landing the fix for Thrift to avoid causing production issues.
Unfortunately almost every such caller has a gflag for it (that are usually have confusing names, e.g. FLAGS_thriftSetMaxNumMessagesInQueue). I’m not touching these gflags, because they may be used in different configs and removing them would cause breakages.
Reviewed By: yfeldblum
Differential Revision: D6129569
fbshipit-source-id: 1550b5073bac42d0c6fb7bdffa1835bf523609c8
Summary:
During the initial phases of a large Buck build, we expect to get a lot of
`glob()` calls during the parse phase (assuming the Buck project is configured
with `glob_handler = watchman`) and a lot of `getSHA1()` calls during the
building phase. Being able to dump this to the log file will make this easier to
audit.
Reviewed By: chadaustin
Differential Revision: D6097764
fbshipit-source-id: 4cb1bb4f6b21830c4d830fcdcf87023eab859f57
Summary:
We have encountered cases where `eden health` reported
`"edenfs not healthy: edenfs not running"` even though the `edenfs` process is
still running. Because the existing implementation of `eden health` bases its
health check on the output of a `getStatus()` Thrift call, it will erroneously
report `"edenfs not running"` even if Eden is running but its Thrift server is
not running. This type of false negative could occur if `edenfs` has shutdown
the Thrift server, but not the rest of the process (quite possibly, its
shutdown is blocked on calls to `umount2()`).
This is further problematic because `eden daemon` checks `eden health`
before attempting to start the daemon. If it gets a false negative, then
`eden daemon` will forge ahead, trying to launch a new instance of the daemon,
but it will fail with a nasty error like the following:
```
I1017 11:59:25.188414 3064499 main.cpp:81] Starting edenfs. UID=5256, GID=100, PID=3064499
terminate called after throwing an instance of 'std::runtime_error'
what(): another instance of Eden appears to be running for /home/mbolin/local/.eden
*** Aborted at 1508266765 (Unix time, try 'date -d 1508266765') ***
*** Signal 6 (SIGABRT) (0x1488002ec2b3) received by PID 3064499 (pthread TID 0x7fd0d3787d40) (linux TID 3064499) (maybe from PID 30644
99, UID 5256), stack trace: ***
@ 000000000290d3cd folly::symbolizer::(anonymous namespace)::signalHandler(int, siginfo_t*, void*)
@ 00007fd0d133cacf (unknown)
@ 00007fd0d093e7c8 __GI_raise
@ 00007fd0d0940590 __GI_abort
@ 00007fd0d1dfeecc __gnu_cxx::__verbose_terminate_handler()
@ 00007fd0d1dfcdc5 __cxxabiv1::__terminate(void (*)())
@ 00007fd0d1dfce10 std::terminate()
@ 00007fd0d1dfd090 __cxa_throw
@ 00000000015fe8ca facebook::eden::EdenServer::acquireEdenLock()
@ 000000000160f27b facebook::eden::EdenServer::prepare()
@ 00000000016107d5 facebook::eden::EdenServer::run()
@ 000000000042c4ee main
@ 00007fd0d0929857 __libc_start_main
@ 0000000000548ad8 _start
Aborted
```
By providing more accurate information to `eden daemon`, if the user tries to
run it while the daemon is already running, they will get a more polite error
like the following:
```
error: edenfs is already running (pid 274205)
```
This revision addresses this issue by writing the PID of `edenfs` in the
lockfile. It updated the implementation of `eden health` to use the PID in the
lockfile to assess the health of Eden if the call to `getStatus()` fails. It
does this by running:
```
ps -p PID -o comm=
```
and applying some heuristics on the output to assess whether the command
associated with that process is the `edenfs` command. If it is, then
`eden health` reports the status as `STOPPED` whereas previously it would report
it as `DEAD`.
Reviewed By: wez
Differential Revision: D6086473
fbshipit-source-id: 825421a6818b56ddd7deea257a92c070c2232bdd
Summary:
Our existing values for these resources appear to be too conservative for
massively parallel builds. Although we expected Eden to behave more like a small
application, there are times when it has to be able to respond more like a
well-provisioned service (particularly during the parse phase of a large Buck
build).
Reviewed By: simpkins
Differential Revision: D6075340
fbshipit-source-id: 7c26e9b0f785358968430527115c63c6d8cdedc8
Summary:
We were previously generating a simple JournalDelta consisting of
just the from/to snapshot hashes. This is great from a `!O(repo)` perspective
when recording what changed but makes it difficult for clients downstream
to reason about changes that are not tracked in source control.
This diff adds a concept of `uncleanPaths` to the journal; these are paths
that we think are/were different from the hashes in the journal entry.
Since JournalDelta needs to be able to be merged I've opted for a simple
list of the paths that have a differing status; I'm not including all of
the various dirstate states for this because it is not obvious how to
reconcile the state across successive snapshot change events.
The `uncleanPaths` set is populated with an initial set of different paths as
the first part of the checkout call (prior to changing the hash), and then is
updated after the hash has changed to capture any additional differences.
Care needs to be taken to avoid recursively attempting to grab the parents lock
so I'm replicating just a little bit of the state management glue in the
`performDiff` method.
The Journal was not setting the from/to snapshot hashes when merging deltas.
This manifested in the watchman integration tests; we'd see the null revision
as the `from` and the `to` revision held the `from` revision(!).
On the watchman side we need to ask source control to expand the list of
files that changed when the from/to hashes are different; I've added code
to handle this. This doesn't do anything smart in the case that the
source control aware queries are in use. We'll look at that in a following
diff as it isn't strictly eden specific.
`watchman clock` was returning a basically empty clock unconditionally,
which meant that most since queries would report everything since the start
of time. This is most likely contributing to poor Buck performance, although
I have not investigated the performance aspect of this. It manifested itself
in the watchman integration tests.
Reviewed By: simpkins
Differential Revision: D5896494
fbshipit-source-id: a88be6448862781a1d8f5e15285ca07b4240593a
Summary:
I noticed that when running `hg push --to master`, the implicit
pull took about a minute to collect all the commits from the remote, then
we tried to make a thrift call to set the parents only to fail with a
python EPIPE stack trace.
I opted to simply make a new client for each call; this is similar to how
things were running with the lame thrift client but doesn't have as terrible
an impact on rebase performance as lame thrift client did.
It feels like things might be better if we made the client retry on transport
errors, but this is quite fiddly so let's defer that until we really need
to cut out the unix domain socket connection overheads.
```
$ hg ci
note: commit message saved in ../.hg/last-message.txt
Traceback (most recent call last):
File "/usr/bin/hg.real", line 47, in <module>
dispatch.run()
File "/usr/lib64/python2.7/site-packages/mercurial/dispatch.py", line 81, in run
status = (dispatch(req) or 0) & 255
File "/usr/lib64/python2.7/site-packages/mercurial/dispatch.py", line 163, in dispatch
ret = _runcatch(req)
File "/usr/lib64/python2.7/site-packages/mercurial/dispatch.py", line 314, in _runcatch
return _callcatch(ui, _runcatchfunc)
File "/usr/lib64/python2.7/site-packages/mercurial/dispatch.py", line 365, in _callcatch
if not handlecommandexception(ui):
File "/usr/lib64/python2.7/site-packages/mercurial/dispatch.py", line 993, in handlecommandexception
ui.log("commandexception", "%s\n%s\n", warning, traceback.format_exc())
File "/usr/lib64/python2.7/site-packages/hgext3rd/sampling.py", line 79, in log
return super(logtofile, self).log(event, *msg, **opts)
File "/usr/lib64/python2.7/site-packages/hgext/blackbox.py", line 163, in log
parents = ctx.parents()
File "/usr/lib64/python2.7/site-packages/mercurial/context.py", line 286, in parents
return self._parents
File "/usr/lib64/python2.7/site-packages/mercurial/util.py", line 862, in __get__
result = self.func(obj)
File "/usr/lib64/python2.7/site-packages/mercurial/context.py", line 1546, in _parents
p = self._repo.dirstate.parents()
File "hgext3rd/eden/eden_dirstate.py", line 163, in parents
File "hgext3rd/eden/eden_dirstate.py", line 157, in _getparents
File "/usr/local/fb-mercurial/eden/hgext3rd/eden/EdenThriftClient.py", line 145, in getParentCommits
parents = self._client.getParentCommits(self._root)
File "facebook/eden/EdenService.py", line 7330, in getParentCommits
File "facebook/eden/EdenService.py", line 7339, in send_getParentCommits
File "thrift/transport/THeaderTransport.py", line 411, in flush
File "thrift/transport/THeaderTransport.py", line 516, in flushImpl
File "thrift/transport/TSocket.py", line 315, in write
thrift.transport.TTransport.TTransportException: Socket send failed with error 32 (Broken pipe)
```
Reviewed By: bolinfest
Differential Revision: D5942205
fbshipit-source-id: 464e5035075476f47232ca975e107e165057c912
Summary:
The eden mount may be in a deep tree (for instance, in the
watchman integration tests) and be too long to fit in the unix domain
socket address.
The failure that we see in that case mistakenly informs the user that
eden is not running, rather than that we have a broken address.
Reviewed By: simpkins
Differential Revision: D5948497
fbshipit-source-id: 3bde8b83bf08b82bdd08758382ac0e218cc12829
Summary:
Originally I thought this would help move towards removing a
`future.get()` call from FileInode, but it turned out to not make a difference
to that code.
It does make it a bit less of a chore to deal with the Journal related diff
callbacks added in D5896494 though, and is a move towards a future where we
could potentially return cached and shared instances of these objects.
This diff is a mechanical change to alter the return type so that we can share
instances returned from the object store interface. It doesn't change any
functionality.
Reviewed By: simpkins
Differential Revision: D5919268
fbshipit-source-id: efe4b3af74e80cf1df20e81b4386450b72fa2c94
Summary: Per wez, this makes the MODIFIED case consistent with the other conflict types (e.g. local_remote). Side benefit of avoiding some naming conflicts in the Haskell/Rust thrift tooling.
Reviewed By: wez
Differential Revision: D5882327
fbshipit-source-id: 3ec68c44d8c8a5c5675f1ced3842d29376d46fe2
Summary:
Following on from D5798659, this diff pulls the mount flow into
EdenServer. Previously the flow of control would bounce back and forth between
the EdenServiceHandler and EdenServer and this made it (IMO) more difficult to
follow and understand where to add things into the flow.
Now `EdenServer::mount` is the main entry point for the mounting flow.
I've simplified the stat registration and broken that out into helper methods
to avoid cluttering up the mount logic.
Reviewed By: bolinfest
Differential Revision: D5806393
fbshipit-source-id: 7c858a2a580332ce82c2600e9dc3537af1d734d1
Summary:
This moves the bind mounting and post-clone script running
functionality to methods of the EdenMount class and makes the whole
mount flow return a `Future<>`. The higher level goal is to make it
easier to see where and how we want to tweak this flow to support
graceful restart.
This is mostly straight forward but care is required to avoid deadlocks; there
are two scenarios:
# We fulfill the fuse start promise in the context of the fuse thread that is
handling the fuse initialization packet before it has signalled to the kernel
that it has come up. This can be solved by using `via(mainEventBase_)`, but...
# When remounting all the mounts on startup, we're running in the
`mainEventBase_` thread so the simplistic solution to 1. would cause us to
deadlock on startup (visible in the remount integration tests).
So to avoid this, we shunt the completion of the future via a CPU pool.
Also worth noting: the way we were setting up the global CPU pool with
wangle wasn't correct; it takes a weak reference to the pool which was
then getting destroyed when our prepare method returned. It just happened
to work for us in the facebook specific build because something else was
setting up a different CPU executor.
I've reconciled this by just setting up a thread pool of our own and
using that explicitly.
Reviewed By: bolinfest
Differential Revision: D5798659
fbshipit-source-id: f1c48730f283f6962f6cd706c02d82ea2952e369
Summary:
This is reasonably straightforward, although a little
more fiddly than I'd hoped because the timer wheel stuff doesn't
offer a convenient way to set up a recurring timer.
I've also made the inode unloading code get run globally for all
mounts; it was previously scheduling one timer per mount point.
This nets out the same; the function scheduler was just a single
thread anyway, so there is no change in the level of concurrency.
I believe that this tidies up the unload counter too; it looked
like we'd set the counter to be the result of the last mount
point that we processed rather than the aggregate of all mounts.
Having the unload timer be associated with the server rather
than the mount points means that we don't have to do anything
special to coordinate with the timer management when the mount
point is being torn down.
Reviewed By: bolinfest
Differential Revision: D5792938
fbshipit-source-id: 1a14bb7b7f4952139e684fe6b52f64bd1ba70dd0
Summary:
Previously, we were generating a bit of disconcerting noise in our logs when
requesting a non-existent key in the dirstate or its copy map. We were also
susceptible to a logical error in the Eden side being silently translated to
a `KeyError` on the Python side.
Now we make things more explicit by converting a `std::out_of_range` on the C++
side to an explicit `NoValueForKeyError` that is defined in `eden.thrift`.
Now the Python side catches a `NoValueForKeyError` explicitly and converts it
into a `KeyError`. Other types of exceptions should pass through rather than be
swallowed.
This also updates the log messages to communicate when a there is no value for a
key. The messaging is improved so that it no longer appears to be a logical
error.
Reviewed By: wez
Differential Revision: D5800833
fbshipit-source-id: c44f2caf04622475d218593037cc6616bbb1c701
Summary: This looks like it ended up getting done together with the original diff
Reviewed By: simpkins
Differential Revision: D5796901
fbshipit-source-id: 24ab05c50b13a37eefe903de5fd3f2dac3d462da
Summary:
After performing the dumb merge of EdenMount and MountPoint in
the prior commit, this one tidies up the state tracking and the interface
by which clients of the object can be notified of state changes.
I've introduced two Promises; the first of these can be used to wait
for the fuse mount to come up or error out. It logically replaces
the cond wait in the `start` method and is exposed to the caller
as a Future, allowing them to wait and react to the outcome.
The second of the promises is associated with the fuse thread pool
winding down. The attached future can be extracted and used by the
client of the EdenMount class. This future yields the fuse device
descriptor which we can then choose to pass on during graceful
restart or simply close. In the current integration, since we ignore
the result of that future, the handle is implicitly closed.
These promises allow us to remove the reference cycle that we had with the
`onStop` function and to potentially make the mount/unmount sequence more
async.
Reviewed By: bolinfest
Differential Revision: D5778214
fbshipit-source-id: 00b293009b7251ddd8bfb10795a115188e97aa3a
Summary:
This is a mechanical and dumb move of the code from MountPoint
and into the EdenMount class.
Of note, it doesn't merge together the two different state/status fields
into a unified thing; that will be tackled in a follow on diff.
Reviewed By: bolinfest
Differential Revision: D5778212
fbshipit-source-id: 6e91a90a5cc760429d87a475ec12f81b93f87be0
Summary:
The higher level goal is to make it easier to deal
with the graceful restart scenario.
This diff removes the SessionDeleter class and effectively renames
the Channel class to FuseChannel. The FuseChannel represents
the channel to the kernel; it can be constructed from a fuse
device descriptor that has been obtained either from the privhelper
at mount time, or from the graceful restart procedure. Importantly
for graceful restart, it is possible to move the fuse device
descriptor out of the FuseChannel so that it can be transferred
to a new eden process.
The graceful restart procedure requires a bit more control over
the lifecycle of the fuse event loop so this diff also takes over
managing the thread pool for the worker threads. The threads
are owned by the MountPoint class which continues to be responsible
for starting and stopping the fuse session and notifying EdenServer
when it has finished. A nice side effect of this change is that
we can remove a couple of inelegant aspects of the integration;
the stack size env var stuff and the redundant extra thread
to wait for the loop to finish.
I opted to expose the dispatcher ops struct via an `extern` to
simplify the code in the MountPoint class and avoid adding special
interfaces for passing the ops around; they're constant anyway
so this doesn't feel especially egregious.
Reviewed By: bolinfest
Differential Revision: D5751521
fbshipit-source-id: 5ba4fff48f3efb31a809adfc7787555711f649c9
Summary:
this is the dumb and obvious refactor of this method to
propagate and wait on the Future from the EdenMount::create call.
Reviewed By: simpkins
Differential Revision: D5750372
fbshipit-source-id: fb7ce595de3bacab99ce8af6ef597ef6f0417c12
Summary: Add a method to get the EventBase used to drive the main thread.
Reviewed By: wez
Differential Revision: D5750054
fbshipit-source-id: ad2eba021a6200ed28e39a60b16d90aabfaee5b4
Summary:
This moves logic for running the server from main.cpp into the EdenServer
class.
This will make it easier to refactor some of the start-up and running process
in the future, and makes EdenServer the owner of this entire workflow. This
will help as we start splitting the startup code into two separate code paths:
one for a new, fresh start, and one for graceful restart taking over mounts
from an existing eden process.
Reviewed By: bolinfest
Differential Revision: D5732656
fbshipit-source-id: 63f05eb1105078764f4e4931d770416dd5f6d6dc
Summary:
Added new tool to report stat information of EdenFs like fuse counters, Memory counters, latencies, Inode status for all the mount points etc.
eden stat : Prints the general information about eden like list of mount points, loaded unloaded and materialized inodes in each mount point. Also this reports how well periodic unload job is doing by reporting the number of unloaded inodes by periodic job.
eden stat io : Prints how many number of calls made to a system call in Edenfs.
eden stat memory : returns the memory stat for edenfs.
eden stat latency : reports the latencies of system calls in Edenfs.
Reviewed By: bolinfest
Differential Revision: D5660345
fbshipit-source-id: 97a1c2b83a6d8df0cd1b82c4d54b52d7ebd126bd
Summary:
This test was supposed to be a part of D5627411 but it was causing strange behaviour so was brought to a separate diff for further investigation.
After investigating, the test didn't pass because the UnloadedInodeData struct only contained the name of the file, not the path to it. The fix for this was to implement a way to get the relative path of the file even after the inode is unloaded.
Reviewed By: simpkins
Differential Revision: D5646929
fbshipit-source-id: f166398a651e8aea49da7e4474a5ad7fde2eaa4e
Summary:
Because `DBG2` seems to be the level we are using to log thrift calls in
`EdenServiceHandler`, this seems like a reasonable default.
Reviewed By: simpkins
Differential Revision: D5686115
fbshipit-source-id: 2d3e0173df37919b6936f73e641f880d16dc539f
Summary:
Note that this feature was mostly implemented before this commit, but never
tested. Unsurprisingly, there were bugs.
This change also introduces a new `eden debug hg_copy_map_get_all` subcommand
because that was a straightforward way to verify the internal state of the copy
map on the server side from an integration test.
Adding this test uncovered a key copy/paste bug in `EdenThriftClient.py`
(`hgCopyMapGet` was being invoked instead of `hgCopyMapPut`.)
It also uncovered a bug in `LameThriftClient` because the `compile()` and
`eval()` calls on the output are not appropriate when the return type of the
Thrift endpoint is `string`.
Reviewed By: simpkins
Differential Revision: D5686114
fbshipit-source-id: f0093d2b67062c01982dc5bc1f0db2774b3a9356
Summary:
1.Modified `TreeInode::unloadChildrenNow()` to return number of inodes that have been unloaded.
2.Modified `EdenServiceHandler::unloadInodeForPath()` to return number of inodes that are unloaded.
Reviewed By: simpkins
Differential Revision: D5627539
fbshipit-source-id: 4cdb0433dced6bf101158b9e6f8c35de67d9abbe
Summary:
Added a test case `test_unload_free_inodes_age` to verify the behaviour of unloadChildrenNow with age parameter.
Added new parameter age to `unloadInodeForPath` in eden.thrift, and `EdenServiceHandler`.
Modified `do_unload_inodes` function in `debug.py` to support the new behaviour.
Reviewed By: simpkins
Differential Revision: D5565859
fbshipit-source-id: a35053725be26bc906cf158969cbe21db1cbadde
Summary:
When Hg tells the `dirstate` to `clear()`, we should also clear out any data we
have on the server for the `Dirstate`.
As it stands, the way we subclass `dirstate`, it does not appear like `clear()`
should be called, in practice, though one thing that could call it is
`hg debugrebuilddirstate`. It is probably good for us to have an RPC lying
around that we can use to reset the `Dirstate.`
Reviewed By: wez
Differential Revision: D5675298
fbshipit-source-id: 38926cfd93f4f83e4c28910f812a693cb32e423a
Summary: This will make subsequent changes to these files cleaner.
Reviewed By: wez
Differential Revision: D5675296
fbshipit-source-id: 06b14d55485415e3ec8a59a4bcc50e6189464b7d
Summary: Provide a thrift interface to invalidate the cache for an inode denoted by path.
Reviewed By: simpkins
Differential Revision: D5655387
fbshipit-source-id: 887aa4963d216a0d8eed93b6fb8721632cc31d19
Summary:
Previously, we were overloading `hgSetDirstateTuple()` to also make it possible
to delete an entry from the internal `hgDirstateTuples` map. Now we have an
explicit method to do this, which enables us to remove some hacks/TODOs.
Reviewed By: simpkins
Differential Revision: D5665170
fbshipit-source-id: bc0753d4990c8966fd9e6c50b29a954d5023292f
Summary:
This was helpful when debugging interactions between Mercurial's
dirstate and the related Thrift calls to Eden server.
Reviewed By: simpkins
Differential Revision: D5664872
fbshipit-source-id: 4e6ef3b034f4fc81f0d467974311a58f54b6e47b
Summary: Modified `TreeInode::unloadChildrenNow` such that inodes are unloaded whose age is greater than a specific age.
Reviewed By: simpkins
Differential Revision: D5526137
fbshipit-source-id: 91e2364d55e31befedcf43d98c26467e1a472ef9
Summary:
Update the EdenServer code to correctly aggregate the thread local stats data
from all threads. Previously it only aggregated stats from the
FunctionScheduler thread that it was running in. This thread never updates any
stats, so it never had any actual stats data.
This also adds a thrift call to trigger stats flushing immediately. This can
be called from integration tests to ensure that they get up-to-date stats
information.
Reviewed By: bolinfest
Differential Revision: D5657267
fbshipit-source-id: 060a24c00a19568b09ae8795477d73a3baab9a3c
Summary:
Update all of the code using ThreadLocal<EdenStats> to pass in a non-default
Tag parameter to the ThreadLocal template.
A non-default tag parameter is required to use the accessAllThreads() method on
the ThreadLocal object. We need to use accessAllThreads() to perform stats
aggregation correctly. Currently the EdenServer code is only performing
aggregation for stats in the FunctionScheduler.
This diff only updates the ThreadLocal<EdenStats> type, but does not contain
any behavior changes. I will fix the stats aggregation in a subsequent diff.
Reviewed By: bolinfest
Differential Revision: D5657268
fbshipit-source-id: bc4b6f56324eb8d3052c023fd3f6f64f99b1d4e0
Summary: Add a command to lookup the path for an inode given the inode number and eden mount path.
Reviewed By: bolinfest
Differential Revision: D5627411
fbshipit-source-id: 25928f506d3f48d8a6784fe81fb17fa0500d6bc9
Summary: It wasn't worth putting these into separate diffs, so just do them all at once.
Reviewed By: yfeldblum
Differential Revision: D5619187
fbshipit-source-id: 7651c062601f4134fac4e4fea654e64d7d8cb8c8
Summary:
We're seeing that this is always set to true for eden,
which is causing buck to run slower than it should.
To make this work correctly, I've augmented our journal data structure
so that it can track create, change and remove events for the various
paths.
I've also plumbed rename events into the journal.
This requires a slightly more complex merge routine, so I've refactored the two
call sites that were merging in slightly different contexts so that they can
now share the same guts of the merge routine. Perhaps slightly
counterintuitive in the merge code is that we merge a record from the past into
the state for now and this is a bit backwards compared to how people think.
I've expanded the eden integration test to check that we don't mix up
create/change/removes for the same path in a given window.
On the watchman side, we use the presence of the filename in the createdPaths
set as a hint that the file is new. In that case we will set the watchman
`ctime` (which is not the state ctime but is really the *created clock time*)
to match the current journal position if the file is new, or leave it set
to 0 if the file is not known to be new. This will cause the `is_new`
flag to be set appropriately by the code in `watchman/query/eval.cpp`;
if the sequence is 0 then it should never be set to true. Otherwise (when
the file was in the `createPaths` set) it will be set to the current journal
position and this will be seen as newer than the `since` constraint on
the query and cause the file to show as `new`.
Reviewed By: bolinfest
Differential Revision: D5608538
fbshipit-source-id: 8d78f7da05e5e53110108aca220c3a97794f8cc2
Summary:
Update EdenServer to use folly::Synchronized for managing access to the
mountPoints_ map, instead of using a separate standalone mutex. This simply
makes it slightly harder to accidentally access the mount point map without
holding the lock correctly.
Reviewed By: bolinfest, wez
Differential Revision: D5541315
fbshipit-source-id: 094a941a3b8177aed4a75b91a998494f4c33fe8c
Summary:
This fixes EdenServer::unmount() to actually wait for all EdenMount cleanup
to complete, and fixes unmountAll() to return a Future that correctly waits for
all mount points to be cleaned up.
Previously `unmount()` waited for the mount point to be unmounted from the
kernel, but did not wait for EdenMount shutdown to complete. Previously
EdenMount shutdown was not triggered until the last reference to the
shared_ptr<EdenMount> was released. This often happened in the FUSE channel
thread that triggered the mountFinished() call--it would still hold a
reference to this pointer, and would not release it until after
mountFinished() returns. As a result, when the main thread was shutting down,
`main()` would call `unmountAll()`, and then return soon after it completed.
Some FUSE channel threads may still be running at this point, still performing
`EdenMount` shutdown while the main thread was exiting. This could result in
crashes and deadlocks as shutdown tried to access objects already destroyed by
the main thread.
With this change `EdenMount::shutdown()` is triggered explicitly during
`mountFinished()`, and `unmount()` will not complete until this finishes.
The `EdenMount` object may still exist at this point, and could still be
deleted by the FUSE channel thread, but the deletion now only requires freeing
the memory and does not require accessing other data that may have been cleaned
up by the main thread.
We should still clean up the FUSE channel thread handling in the future, to
make sure these threads are joined before the main thread exits. However, that
cleanup can wait until a separate diff. Ideally I would like to move more of
the mount and unmount logic from EdenServer and EdenServiceHandler and put that
code in EdenMount instead.
Reviewed By: bolinfest
Differential Revision: D5541318
fbshipit-source-id: 470332478357a85c314bc40458373cb0f827f62b
Summary:
When cleaning up an unmounted EdenMount, cancel the periodic unload inodes
function before we remove the mount from the mountPoints_ map and signal other
threads that it has been removed via mountPointsCV_. Also use
FunctionScheduler::cancelAndWait() to make sure that we wait for the function
to complete if it was currently in the middle of being run.
Reviewed By: bolinfest
Differential Revision: D5541316
fbshipit-source-id: 46046f59746b5f283a5d4af94159b5c122d43d0d
Summary:
Make all the changes in thrift to enable the use of mstch-cpp2.
For now, if any cpp2 flag is present, it will fallback to the python genenerator (i.e. t_cpp_generator.py)
Reviewed By: yfeldblum
Differential Revision: D5464216
fbshipit-source-id: 287618fbf3b2dcc3142e305e648e48b1ebb762c1
Summary:
Added `EdenServer::unloadInodesScheduler_` to schedule unloading inodes on timely basis.
new function will be added to the function scheduler on each mount, so each mount point can be have different start delay and different frequency.
Reviewed By: simpkins
Differential Revision: D5501036
fbshipit-source-id: 238b9cb5747a2358c65a1508095ad672bdf87ffc
Summary:
Some test cases are crashing once in a whiled due to a race condition on lock over overlay directory.
when a mount point is unmounted, `EdenServer::mountFinished` called on `umount` where destruction of `EdenMount` object for the mount point happens. When a remount is called before the destruction of `EdenMount` a race condition is occuring on the lock of overlay directory as the undestructed `EdenMount` still holds the lock on overlay and the newly created `EdenMount` tries to grab a lock on overly.
Made the `EdenServer::unmount` asynchronous which waits for a promise that is set in `EdenServer::mountFinished`.
Reviewed By: simpkins
Differential Revision: D5515740
fbshipit-source-id: 5cbfddf13fe00c2cb1b63a6be6353e9b84fbd569
Summary:
Move the dropPrivileges() function from the PrivHelper code to the UserInfo
class, and update it to correctly set supplementary groups when dropping
privileges.
Reviewed By: bolinfest
Differential Revision: D5501254
fbshipit-source-id: 62d7a91685ae72a73e848236f6e6f636b6203481
Summary:
This adds a new UserInfo class to store the UID, GID, as well as the username
and home directory. This moves the determineUid() and determineGid() functions
from main.cpp into this new class, and makes the logic somewhat smarter now.
In addition to looking up the UID and GID, we now look up the username. This
information is not used yet, but will be used in an upcoming diff to set
supplementary groups.
This also stores the home directory in the UserInfo class. The home directory
is usually necessary to find the user's ~/.edenrc file. Computing it as part
of UserInfo makes the most sense since we will likely have already looked up
the user's passwd entry.
Reviewed By: bolinfest
Differential Revision: D5501252
fbshipit-source-id: 1cb4be9f6c1493de4362da3393034e78bedd9db2
Summary:
This is a relatively simple test that takes two branches with no conflicts and
rebases one on top of the other. It also provides modest checks to ensure Eden
does not load a bunch of inodes unnecessarily when updating to the new head.
This also introduces `EdenServerInspector`, which provides convenience methods
for inspecting the Eden server via Thrift.
Reviewed By: simpkins
Differential Revision: D5504741
fbshipit-source-id: 6636c431658f24a850d0e5404d1a0e4f0528a781
Summary:
D5483953 added a check to ensure that getMode() can only be called on entries
that do not have a loaded inode. However, a few places in the code were still
calling getMode() on tree entries without checking if the inode was loaded or
not.
These crashes were caught in the integration tests run by `buck test eden/...`,
but were not caught by sandcastle tests on the original diff since sandcastle
only runs the eden unit tests, and not the integration tests.
All of these locations only needed to check the file type, which is safe to do
even if the inode is loaded, since the file type portion of the mode can never
change on an existing inode. Only the permissions bits are unsafe to access
once an inode has been loaded (since we need to ask the inode itself for the
latest permissions bits).
I updated these call sites to stop using getMode() and instead use functions
that explicitly check only the file type bits.
Reviewed By: bolinfest
Differential Revision: D5501256
fbshipit-source-id: c989dab2fdacb5b9cdecb6f5101795298f57e78b
Summary:
Before this commit, TreeInode::Entry was a struct which had two
private members: mode and inode. In this commit,
1. TreeInode::Entry was changed from struct to class.
2. Appropriate getters and setters were introduced for the public members to
make them private.
3. Existing code accessing the public members directly was modified to use
the getters instead.
4. A couple of TODOs were added to address Overlay::saveOverlayDir()'s access
of child inode information.
Reviewed By: simpkins
Differential Revision: D5483953
fbshipit-source-id: 50d526731e193a3a4a32742bd4d49deb9ee6b432
Summary:
- Updated makeShared() to return a folly::Future<std::shared_ptr<EdenMount>> instead of just a std::shared_ptr<EdenMount>.
- Updated callers to use the returned future (get() for now)
- Refactor makeShared() and the EdenMount constructor to avoid blocking.
Reviewed By: simpkins
Differential Revision: D5424088
fbshipit-source-id: f026a3a3e4abb3593bafda76673e12c55da26322
Summary:
Update EdenServer to catch SIGTERM and SIGINT, and to stop gracefully on these
signals.
Reviewed By: wez
Differential Revision: D5315323
fbshipit-source-id: 16611190c8d522a78cf6b624d97d4f7ecc478f96
Summary: This includes adjusting them to conform to Folly's style for single file sources and headers, making targets as small as possible, not providing aggregate targets, and using auto-deps.
Reviewed By: yfeldblum
Differential Revision: D5312078
fbshipit-source-id: 1c7f5aa04da3bad236ffa23000c7bda47b82e3ef
Summary:
Update HgBackingStore and GitBackingStore to accept the repository path as an
AbsolutePathPiece rather than as a plain StringPiece.
Reviewed By: bolinfest
Differential Revision: D5309172
fbshipit-source-id: aca4818c3add11d07ece796298f6175ca4fb1448
Summary:
1. Moved read, write, mkdir, rm methods in hg/lib/hg_extension_test_base.py to lib/test_case.py.
2. Added integration test case to test unload free inodes.
Reviewed By: simpkins
Differential Revision: D5277870
fbshipit-source-id: b93b6049a10357cf8c92366e6dca3968f7f30c30
Summary:
Update eden to log via the new folly logging APIs rather than with glog.
This adds a new --logging flag that takes a logging configuration string.
By default we set the log level to INFO for all eden logs, and WARNING for
everything else. (I suspect we may eventually want to run with some
high-priority debug logs enabled for some or all of eden, but this seems like a
reasonable default to start with.)
Reviewed By: wez
Differential Revision: D5290783
fbshipit-source-id: 14183489c48c96613e2aca0f513bfa82fd9798c7
Summary:
Now that the non-future versions of these APIs have been removed, rename
getBlobFuture() to getBlob(), and getTreeFuture() to getTree()
Reviewed By: wez
Differential Revision: D5295690
fbshipit-source-id: 30dcb88854b23160692b9cd83a632f863e07b491
Summary:
1.Added a new thrift method to unload free inodes of a directory and its sub directories.
2.Added a new debug sub command 'eden debug unload <path>' to cli tools to unload free inodes.
Reviewed By: simpkins
Differential Revision: D5261038
fbshipit-source-id: 85b4c5ae18c0ae24c666a44ac9892765e753397f
Summary:
This is a major change to Eden's Hg extension.
Our initial attempt to implement `edendirstate` was to create a "clean room"
implementation that did not share code with `mercurial/dirstate.py`. This was
helpful in uncovering the subset of the dirstate API that matters for Eden. It
also provided a better safeguard against upstream changes to `dirstate.py` in
Mercurial itself.
In this implementation, the state transition management was mostly done
on the server in `Dirstate.cpp`. We also made a modest attempt to make
`Dirstate.cpp` "SCM-agnostic" such that the same APIs could be used for
Git at some point.
However, as we have tried to support more of the sophisticated functionality
in Mercurial, particularly `hg histedit`, achieving parity between the clean room
implementation and Mercurial's internals has become more challenging.
Ultimately, the clean room implementation is likely the right way to go for Eden,
but for now, we need to prioritize having feature parity with vanilla Hg when
using Eden. Once we have a more complete set of integration tests in place,
we can reimplement Eden's dirstate more aggressively to optimize things.
Fortunately, the [[ https://bitbucket.org/facebook/hg-experimental/src/default/sqldirstate/ | sqldirstate ]]
extension has already demonstrated that it is possible to provide a faithful
dirstate implementation that subclasses the original `dirstate` while using a different
storage mechanism. As such, I used `sqldirstate` as a model when implementing
the new `eden_dirstate` (distinguishing it from our v1 implementation, `edendirstate`).
In particular, `sqldirstate` uses SQL tables as storage for the following private fields
of `dirstate`: `_map`, `_dirs`, `_copymap`, `_filefoldmap`, `_dirfoldmap`. Because
`_filefoldmap` and `_dirfoldmap` exist to deal with case-insensitivity issues, we
do not support them in `eden_dirstate` and add code to ensure the codepaths that
would access them in `dirstate` never get exercised. Similarly, we also implemented
`eden_dirstate` so that it never accesses `_dirs`. (`_dirs` is a multiset of all directories in the
dirstate, which is an O(repo) data structure, so we do not want to maintain it in Eden.
It appears to be primarily used for checking whether a path to a file already exists in
the dirstate as a directory. We can protect against that in more efficient ways.)
That leaves only `_map` and `_copymap` to worry about. `_copymap` contains the set
of files that have been marked "copied" in the current dirstate, so it is fairly small and
can be stored on disk or in memory with little concern. `_map` is a bit trickier because
it is expected to have an entry for every file in the dirstate. In `sqldirstate`, it is stored
across two tables: `files` and `nonnormalfiles`. For Eden, we already represent the data
analogous to the `files` table in RocksDB/the overlay, so we do not need to create a new
equivalent to the `files` table. We do, however, need an equivalent to the `nonnormalfiles`
table, which we store in as Thrift-serialized data in an ordinary file along with the `_copymap`
data.
In our Hg extension, our implementation of `_map` is `eden_dirstate_map`, which is defined
in a Python file of the same name. Our implementation of `_copymap` is `dummy_copymap`,
which is defined in `eden_dirstate.py`. Both of these collections are simple pass-through data
structures that translate their method calls to Thrift server calls. I expect we will want to
optimize this in the future via some client-side caching, as well as creating batch APIs for talking
to the server via Thrift.
One advantage of this new implementation is that it enables us to delete
`eden/hg/eden/overrides.py`, which overrode the entry points for `hg add` and `hg remove`.
Between the recent implementation of `dirstate.walk()` for Eden and this switch
to the real dirstate, we can now use the default implementation of `hg add` and `hg remove`
(although we have to play some tricks, like in the implementation of `eden_dirstate.status()`
in order to make `hg remove` work).
In the course of doing this revision, I discovered that I had to make a minor fix to
`EdenMatchInfo.make_glob_list()` because `hg add foo` was being treated as
`hg add foo/**/*` even when `foo` was just a file (as opposed to a directory), in which
case the glob was not matching `foo`!
I also had to do some work in `eden_dirstate.status()` in which the `match` argument
was previously largely ignored. It turns out that `dirstate.py` uses `status()` for a number
of things with the `match` specified as a filter, so the output of `status()` must be filtered
by `match` accordingly. Ultimately, this seems like work that would be better done on the
server, but for simplicity, we're just going to do it in Python, for now.
For the reasons explained above, this revision deletes a lot of code `Dirstate.cpp`.
As such, `DirstateTest.cpp` does not seem worth refactoring, though the scenarios it was
testing should probably be converted to integration tests. At a high level, the role of
`DirstatePersistence` has not changed, but the exact data it writes is much different.
Its corresponding unit test is also disabled, for now.
Note that this revision does not change the name of the file where "dirstate data" is written
(this is defined as `kDirstateFile` in `ClientConfig.cpp`), so we should blow away any existing
instances of this file once this change lands. (It is still early enough in the project that it does
not seem worth the overhead of a proper migration.)
The true test of the success of this new approach is the ease with which we can write more
integration tests for things like `hg histedit` and `hg graft`. Ideally, these should require very
few changes to `eden_dirstate.py`.
Reviewed By: simpkins
Differential Revision: D5071778
fbshipit-source-id: e8fec4d393035d80f36516ac050cad025dc3ba31
Summary:
We're using `window` to constrain resources and maintain an upper
bound on the number of futures that we spawn. Ironically, for a largish number
of already completed futures, this results in a recursive chain of `spawn`
calls in the `window` implementation and results in a stack overflow (the kind
that crashes a program, not the kind that answers programming questions).
This diff removes the use of `window` and replaces it with `collect`.
An earlier iteration of this diff just made all of these calls serially,
blocking the main thread. This at least does things in parallel; both
can torture the system for pathological walks, but at least the parallel
one should complete faster.
Reviewed By: simpkins
Differential Revision: D4932774
fbshipit-source-id: 40d39a85029f38ff69a530070efb879a81950451
Summary:
Update the eden thrift APIs to support multiple thrift parents, and update the
hg extension to use the new thrift APIs.
Reviewed By: bolinfest
Differential Revision: D4949139
fbshipit-source-id: 98a7bd17ccad7be6a429df34ecfaf3fe7ae0b39a
Summary:
This updates the ClientConfig and EdenMount code to support storing two parent
commits.
This changes the on-disk SNAPSHOT file contents add an 8-byte header that
includes a file identifier and a file format version number, followed by up to
two commit hashes. The code currently can read either the old or new format
from the SNAPSHOT file. We should be able to drop the code for reading the old
format fairly soon if we want, though.
This diff only updates the ClientConfig and EdenMount code, and does not yet
update the thrift APIs or the eden mercurial extension yet. I will update the
rest of the code in a subsequent diff.
Reviewed By: bolinfest, wez
Differential Revision: D4943917
fbshipit-source-id: cf456e67b845aa0cf8b45c822985cb932df107b4
Summary:
This fixes "hg commit" so that it correctly updates the in-memory snapshot.
This has been broken ever since I added the in-memory snapshot when
implementing checkout(). The existing scmMarkCommitted() method updated only
the Dirstate object and the on-disk SNAPSHOT file.
This diff fixes checkout() and resetCommit() to clear the Dirstate user
directives correctly, and then replaces calls to scmMarkCommitted() with
resetCommit().
Reviewed By: bolinfest
Differential Revision: D4935943
fbshipit-source-id: 5ffcfd5db99f30c730ede202c5e013afa682bac9
Summary:
This tweaks the main.cpp and EdenServer.cpp code so that we can avoid having
separate internal and open source builds of the eden/fs/service:server library.
Only one small function is different between the two builds. This updates the
code so that this function is only used from main.cpp, instead of being used
inside EdenServer.cpp.
Reviewed By: wez
Differential Revision: D4889721
fbshipit-source-id: 97abda92c36752516fb8a8e73b4cf50580db8d79
Summary:
This change makes it so that all of the C++ code related to the edenfs daemon
is now contained in the eden/fs subdirectory.
Reviewed By: bolinfest, wez
Differential Revision: D4889053
fbshipit-source-id: d0bd4774cc0bdb5d1d6b6f47d716ecae52391f37
Summary:
This is a bit too magical -- it's not clear that the thing produces an Options object. If you do know that you can chain further option setters off this thing, it's nice, but otherwise, the first impression is "what just happened?".
So, let's have one good way for doing things.
Reviewed By: yfeldblum
Differential Revision: D4863947
fbshipit-source-id: 3dfe83cfc077d47f604f47dcb21149fbaa2d2243
Summary:
Update the hg extension to implement dirstate.rebuild(). This is necessary for
the `hg reset` command. This also now implements dirstate.setparents() for
cases when there is only one parent.
Reviewed By: wez
Differential Revision: D4823780
fbshipit-source-id: 802de006e03860995095dc3af17acb2eb05f4e8b
Summary:
Update the checkOutRevision() thrift API to accept the commit ID as a
BinaryHash rather than as a hexadecimal ASCII string, to make it more
consistent with our other thrift APIs.
Reviewed By: wez
Differential Revision: D4823779
fbshipit-source-id: 6226f926533ef86d60f8b1d659bc5d912e2855e6
Summary:
This avoids hard coding knowledge of the FB-specific `local` dir.
This is only suitable for use on already-mounted eden instances,
so the function accepts either the mounted path or the eden
configuration dir.
Reviewed By: bolinfest
Differential Revision: D4829166
fbshipit-source-id: df9daed6024cdec6222be90f5cb624dd9ca0da0b
Summary:
Folly is dropping support for GCC 4.8 now that the branch for last HHVM release that needs to support it has been cut.
This codemod's away all the trivially identifiable uses of `folly::make_unique` in fbcode outside of folly, mcrouter and hphp.
A future diff will kill it in those, and a diff after that will kill it entirely.
Reviewed By: djwatson
Differential Revision: D4498157
fbshipit-source-id: 5a2c05ffd29e5358298acb53dab189a699b022cc
Summary:
This diff adds a new "eden debug" command, with a few subcommands:
- "eden debug inode": Report information about currently loaded inodes
- "eden debug tree": Report data about source control Tree objects
- "eden debug blobmeta": Report metadata about source control Blob objects
- "eden debug blob": Report the contents of source control Blob objects
This diff also includes the thrift APIs in edenfs to support these commands.
Reviewed By: bolinfest
Differential Revision: D4760528
fbshipit-source-id: 7dc2bd6e0e952ba6fb61702c672fb9417d603ffa
Summary:
Previously EdenServer::getMount() returned a null pointer if the path did not
refer to a known mount point. However, most of the thrift APIs using this
method never checked if the return value was null, which could allow any thrift
client to crash edenfs.
This updates getMount() to throw if the path is not found, and never return
null.
Reviewed By: bolinfest
Differential Revision: D4760527
fbshipit-source-id: 6cd9a1e3bc0a0a220d7f5fa518d497d37f245eff
Summary:
Tweaks the stats stuff so that we can name the histograms and export them with p50, p90, p99 percentiles.
This is made a bit ugly because the fb303 stats code isn't yet open source, so
there's a moderately ugly preprocessor directive that we assume to be true for
our internal build. We'll need to force that to be 0 for the open source build
until we open the stats code and make it available.
Reviewed By: bolinfest
Differential Revision: D4801868
fbshipit-source-id: 643909e63bd4a74b2cfa580be131f65c5673bc94
Summary:
This is an initial pass at implementing "hg status" using the new EdenMount and
TreeInode diff() code.
More work still needs to be done to clean things up here, but this makes
"hg status" more correct in the face of modified files and directories that are
not modified, and vice-versa.
In particular, the following still needs to be cleaned up in future diffs:
- Update the "hg status" code to more correctly process user directives for
paths that did not appear in the diff output. This will probably be
simplified some by correctly updating the user directive list on checkout and
resetCommit() operations. It may also make things simpler if we store the
user directives in a hierarchical path map, rather than in a flat list.
- Update the addAll() code to also use the new diff logic, rather than the
getStatusForExistingDirectory() function.
- Clean up the locking some. I think it may be worth changing the code to use
a single lock to manage both the current commit ID for the mount and the
dirstate user directives. We probably should hold this lock for the duration
of a diff operation. The diff operation should also return the commit ID
that it is based against.
Reviewed By: wez
Differential Revision: D4752281
fbshipit-source-id: 6a6d7e2815321b09c75e95aa7460b0da41931dc0
Summary:
It's not really magic because we don't have a virtual directory
inode base any more. Instead, we mkdir and populate it at mount time.
What is slightly magical about it is that we give it some special powers:
* We know the inode number of the eden dir and prevent unlink operations
on it or inside it.
* The .eden dir is present in the contents of the root inode and will
show up when that directory is `readdir`'d
* When resolving a child of a TreeInode by name, we know to return the
magic `.eden` inode number. This means that it is possible to `stat`
and consume the `.eden` directory from any directory inside the eden
mount, even though it won't show up in `readdir` for those child dirs.
The contents of the `.eden` dir are:
* `socket` - a symlink back to the unix domain socket that our thrift
server is listening on. This means that it is a simple
`readlink(".eden/socket")` operation to discover both whether a directory
is part of an eden mount and how to talk to the server.
* `root` - a symlink back to the root of this eden mount. This allows
using `readlink(".eden/root")` as a simple 1-step operation to find
the root of an eden mount, and avoids needing to walk up directory
by directory as is the common pattern for locating `.hg` or `.git`
dirs.
Reviewed By: simpkins
Differential Revision: D4637285
fbshipit-source-id: 0eabf98b29144acccef5c83bd367493399dc55bb
Summary:
This tweaks the definition of the subscribe method in the
streamingeden thrift file and implements the server side of the
thrift service, and the client side to consume it in watchman.
The definition is now a bit simpler than it was previously; we're
now just sending a stream of the updated JournalPosition objects
to the client rather than computing and sending FileDelta objects.
This is cheaper for Eden to deal with because it is very cheap to take
the current journal position and pass that over the wire. This is
important because a burst of mutations will likely queue up a bunch
of these in quick succession.
In a future diff I'm considering adding a latency parameter for
the subscription so that we can constrain the number of updates
over a certain time period (likely in the 10's of milliseconds range).
For now I just want to prove that the concept works.
On the watchman side we just need to pull these off the wire as they are sent
by eden, then wait for the subscription stream to settle before internally
broadcasting to any connected subscribers.
Reviewed By: bolinfest
Differential Revision: D4647259
fbshipit-source-id: 03aa16e59a43195a2505a8d03bce1ccf88a8d42f
Summary:
Update the edenfs main function to check the exit code of the privhelper
process, and to exit with EX_SOFTWARE rather than EX_OK if the privhelper did
not exit with status 0.
This will ensure that our integration tests always catch crashes in the
privhelper process. Previously we could catch errors that happened before the
privhelper socket was closed, but if the privhelper server crashed during
cleanup we never noticed the error.
Reviewed By: bolinfest
Differential Revision: D4728006
fbshipit-source-id: 2c478612e5fabec90438b28b9c76a3343f561655
Summary:
Add a unit test that modifies a file and then does a forced checkout to revert
back to the original commit state.
This also includes some pretty-printers to make the tests print
CheckoutConflict objects nicely if any test assertions fail.
Reviewed By: bolinfest
Differential Revision: D4727779
fbshipit-source-id: b31dc90be28d17ed0f2a4076c96a08ba82588190
Summary:
Update the TreeInode and CheckoutAction to correctly be able to remove
directories during checkout operations. This handles both removing directories
that are newly empty as a result of the checkout, and directories that are
replaced with a file or symlink.
This also refactors the rmdir/unlink code a bit so we can share some of its
logic for doing directory removal during the checkout operation.
Reviewed By: bolinfest
Differential Revision: D4708607
fbshipit-source-id: 3b9dd9cc3536845dad0d9278e3c5d3ac1ed04570
Summary:
Refactor FileInode and FileData so that they no longer store the
TreeInode::Entry that refers to this file. The TreeInode::Entry is owned by
the parent TreeInode and is not safe to access without holding the TreeInode
contents_ lock, which the FileInode code never did. Additionally, once a
FileInode is unlocked the TreeInode::Entry is destroyed. Previously the
FileInode code would retain its pointer to this now invalid data, and could
still dereference it.
In the future I think it might be worth refactoring the FileInode/FileData
split a bit more, but this at least addresses the issues around invalid
accesses to entry_.
Reviewed By: bolinfest
Differential Revision: D4688058
fbshipit-source-id: 17d5d0c4e756afc6e3c4bb279cb1bb5231f3134f
Summary:
Update the DEFS and .buckconfig files so that the C++ code can be built
successfully with buck in our github repository.
Reviewed By: wez
Differential Revision: D4633749
fbshipit-source-id: a07a6e664471237fcc2d13a8be8b92efa3a101bf
Summary:
In order to send realtime subscription information to watchman
we need to use the thrift streaming data functionality. Since this
isn't supported in java we originally broke this out into its own
thrift file.
It's now time to hook this up; this diff adds some plumbing so that
our thrift server knows to implement this interface.
This was made a bit more complicated because the inheritance in
the thrift definitions causes some ambiguities in the header maps
that buck uses to manage the includes.
I also had to rename the file from `eden.stream.thrift` to `streamingeden.thrift`
to avoid a problem with the period in the middle of the identifier.
For the server side of things, I've removed the `:thrift-cpp2` target
and replaced it with `thrift-streaming` instead.
This diff doesn't implement the endpoint, it is just setting things
up for the follow-on diff that does.
Reviewed By: bolinfest
Differential Revision: D4640639
fbshipit-source-id: b3625b0ff33a42945bf523944beed050f549a18c
Summary:
I originally added this to facilitate `hg status` but we didn't end up
using it. Rather than keeping it around and updating the tests in the `.eden`
dir diffs that follow, let's just remove it.
Reviewed By: simpkins
Differential Revision: D4610768
fbshipit-source-id: 158c0207f88980e86aeeddf75e6fd49763d2a402
Summary:
Add a thrift call to get the ID of the snapshot currently checked out in a
given eden mount.
Reviewed By: bolinfest
Differential Revision: D4570729
fbshipit-source-id: 208ceb309aacfb07e30ddb8fdfc0950a3d7d748b
Summary:
This is the initial code for implementing checkout.
This isn't quite 100% implemented yet, but I think it's worth checking in this
code as-is, and getting the remaining functionality done in separate diffs.
In particular, a few operations aren't implemented:
- Removing a directory that was deleted in the new revision
- Replacing a directory that was replaced with a file or symlink in the new
revision
- When doing a forced update, replacing a file or directory that did not exist
in the old revision, but that was created locally in the working directory,
and also exists in the new revision.
Reviewed By: wez
Differential Revision: D4538516
fbshipit-source-id: 5bb4889b02f23ab2048fcae2c8b7614340181aa6
Summary:
Update PrivHelperServer::bindUnmount() to give up after 2 seconds instead of
possibly looping forever.
Also fix the unmount code in the eden process to rethrow the original exception
as-is. Previously it was re-throwing it as a std::exception, losing the
exception details.
I think there are still other bugs here in this unmount code--it repeatedly
hangs for me on unmount. (I believe it is trying to unmount things that aren't
actually bind mounts.) However, I'll look into fixing those in a separate
diff.
Reviewed By: wez
Differential Revision: D4547711
fbshipit-source-id: af6bce0262e8314361450874ef998ebf7ad590e3
Summary:
Refactor the Overlay code to store data using inode numbers rather than the
affected file's path in the repository. This simplifies the TreeInode code a
bit, as we no longer have to rename overlay files to stay in sync with the file
paths. This also eliminates some crashes when trying to update overlay files
for inodes that have been unlinked (and hence no longer have a path). This
also includes a few fixes to avoid writing journal entries for unlinked files
too. Additionally this contains a few fixes to how mode bits are stored in the
overlay, and fixes a bug where create() was ignoring the mode argument.
Reviewed By: wez
Differential Revision: D4517578
fbshipit-source-id: c1e31497dcf62c322b0deff72b0a02675b0509ab
Summary:
Update the code to release the parent TreeInode's contents lock when processing
its children. This makes sure we do not hold the root lock for the entire
duration of this function.
Reviewed By: wez
Differential Revision: D4471289
fbshipit-source-id: dc4b50bd06fcceb7a7df3d7dd04cfc0d41285556
Summary:
This is to facilitate the watchman integration and draws on the
watchman glob implementation; the approach is to split the glob strings into
path components and evaluate the components step by step as the tree is walked.
Components that do not include any glob special characters can be handled as
a direct lookup from the directory contents (O(1) rather than O(num-entries)).
The glob method returns a set of filenames that match a list of
of glob patterns.
Recursive globs are supported. It is worth noting that a glob like "**/*" will
return a list of every entry in the filesystem. This is potentially expensive
and should be avoided. simpkins is in favor of disallowing this as a forcing
function to encourage tool-makers to adopt patterns that don't rely on a
complete listing of the filesystem.
For now I'd like to get this in without such a restriction; it's also worth
noting that running `find .` in the root of the mount point has a similar
effect and we can't prevent that from happening, so the effect of the overly
broad glob is something that we need to be able to withstand in any case.
Unrestricted recursive globs will make it easier to connect certain watchman
queries in the interim, until we have a more expressive thrift API for walking
and filtering the list of files.
Note: I've removed the wildmatch flags that I'd put in the API when I stubbed
it out originally. Since this is built on top of our GlobMatcher code and that
doesn't have those flags, I thought it would be simplest to just remove them.
If we find that we need them, we can figure out how to add them later.
Also Note: the evaluation of the glob is parallel-ready but currently limited
to 1 at a time by constraining the folly::window call to 1. We could make this
larger but would need a more intelligent constraint. For example, a recursive
glob could initiate N concurrent futures per level where N is the number of
sub-dirs at a given level. Using a custom Executor for these futures may be a
better option to set an upper bound on the number of concurrent jobs allowed
for a given glob call.
Depends on D4361197
Reviewed By: simpkins
Differential Revision: D4371934
fbshipit-source-id: 444735600bc16d2c2185f2277ddc5b51f672600a
Summary:
As part of the mount process, make sure we load InodeBase objects for all files
in the mount that are already materialized.
Other parts of the code assume that InodeBase objects are always loaded for
materialized files. We never unload materialized inodes, but if the mount
point was unmounted then remounted we previously did not ensure to load the
materialized InodeBase objects. This diff makes sure we load all materialized
inodes before starting the mount.
Reviewed By: bolinfest
Differential Revision: D4461193
fbshipit-source-id: 70d06fd01e2df333ce2816d5d7a392b0a5d6e1e6
Summary:
Rename the current EdenMount::getInodeBase() function to
EdenMount::getInodeBaseBlocking() and add a new getInodeBase() function that
performs the lookup asynchronously and returns a Future<InodePtr>.
This is implemented with a TreeInode::getChildRecursive() function that allows
asynchronous recursive lookups at any point in the tree.
Reviewed By: bolinfest
Differential Revision: D4427855
fbshipit-source-id: aca55e681a48c848b085b7fc6a13efe6cf0a4e3a
Summary:
While testing with the fb-eden rpm installed, I hit some integration
test failures. These were caused by the integration tests picking up the
default post-clone hook configuration.
This diff changes our existing `systemConfigDir` option (which defaults to
`/etc/eden/config.d`) to `etcEdenDir` (which defaults to `/etc/eden`) and
adjusts the code that consumed `systemConfigDir` to construct the effective
value by computing `etcEdenDir + "config.d"`.
Doing this allows us to also default the `repoHooks` path to be
`etcEdenDir + "hooks"` rather than just hard coding `/etc/eden/hooks`.
The result of this is that our integration tests will now pass when `fb-eden`
is installed, because they override the `etcEdenDir` option and isolate their
`edenfs` processes from the globally installed configuration.
Reviewed By: bolinfest
Differential Revision: D4446321
fbshipit-source-id: 524fdb2f386fdf16dce42dce7661d07e13c0f0e7
Summary:
It's been bothering me that we claim to run and complete running
the globally installed hooks when they aren't there, so tidy up the logging.
Reviewed By: bolinfest
Differential Revision: D4446318
fbshipit-source-id: acda223e1df2302602ad9967b1aa1b334b8477c7
Summary:
This adopts the new InterpolatedPropertyTree class as
the configuration storage.
It doesn't introduce any actual configuration of the interpolation
replacements; that will be in a follow-on diff.
Reviewed By: bolinfest
Differential Revision: D4444157
fbshipit-source-id: 18ead8e9074d23b1154e81f012f0c90efced1350
Summary:
Fix issues flagged by running "arc lint" on all eden files. There are still 6
warnings outstanding, but these are either false positives or advice that we
intentionally are ignoring for legitimate reasons.
Reviewed By: bolinfest
Differential Revision: D4446615
fbshipit-source-id: 992f3c146f99d63935f849aa775dd6d611a04acf
Summary:
Update copyright statements to "2016-present". This makes our updated lint
rules happy and complies with the recommended license header statement.
Reviewed By: wez, bolinfest
Differential Revision: D4433594
fbshipit-source-id: e9ecb1c1fc66e4ec49c1f046c6a98d425b13bc27
Summary:
Update the getSHA1() thrift handler to get the file SHA1 values in parallel.
The inode lookup itself still happens serially at the moment. We need to
provide a future-based version of EdenMount::getFileInode() in the future, and
change all existing callers of it to use the Future-based version.
Reviewed By: wez
Differential Revision: D4361091
fbshipit-source-id: 1abbc16df8c3edf52959a82f16a7f59e5d6d038b
Summary:
Update EdenMount so that it cannot be destroyed directly, and must be destroyed
through a special destroy() function. This function is not implemented yet,
but it will delay actual destruction of the EdenMount until all Inode objects
in the mount have been destroyed.
This diff primarily updates the users of EdenMount to call destroy() properly.
I will send a subsequent diff that implements destroy() at the same time as
implementing Inode unloading.
Reviewed By: wez
Differential Revision: D4360558
fbshipit-source-id: 202826b63b75e1de2b73270806da094206108a47
Summary: Folly will soon be dropping support for GCC 4.8, which means that `folly::make_unique` will be going away, so codemod away as much as possible before then.
Reviewed By: yfeldblum
Differential Revision: D4411221
fbshipit-source-id: 31d61425f6595d0750ea2e4c95c7d42bb5a5d955
Summary:
This updates all of the eden code to use the new InodeMap class. This replaces
the InodeNameManager class and the unordered_map previously stored in the
EdenDispatcher.
Reviewed By: bolinfest
Differential Revision: D4325750
fbshipit-source-id: d80ae7581ba79ca2b63155e184995a3e83e85dc1
Summary:
We can use `//` exclusively because we always build Eden with Buck and never
fbbuild, our legacy build system for fbcode.
This revision was initially created by running:
```
find eden -name TARGETS | xargs sed -i -e 's#@/#//#g'
```
And then manually updating the `DEFS` file now that we no longer need
some normalization code for an outdated pattern.
But then I got annoyed by other inconsistencies, so I went through and
alpha-sorted some lists, replaced all double quotes with single quotes,
and fixed indents to be two spaces.
Reviewed By: simpkins
Differential Revision: D4356724
fbshipit-source-id: ab07a48f12fa937c257213d12331efdf09e42da6
Summary:
Most of the work for this diff was achieved via:
```
find eden -type f | xargs sed -i -e 's#ThriftHgStatusCode#StatusCode#g'
find eden -type f | xargs sed -i -e 's#HgStatusCode#StatusCode#g'
```
Reviewed By: simpkins
Differential Revision: D4237975
fbshipit-source-id: 2ee04a89101291c8972ac7bd3ff6cca92cbb0799
Summary:
This should make the common case of `hg add` or `hg add .` much more efficient
because it no longer performs a walk of the entire repository from the client
side.
Reviewed By: simpkins
Differential Revision: D4317871
fbshipit-source-id: 7061553fe0de0c4afa84b4f835316965088675e8
Summary:
This updates the EdenServer and LocalStore classes to require more arguments be
passed in as AbsolutePath arguments rather than just plain strings.
This updates the main program to process path arguments using canonicalPath().
Reviewed By: bolinfest
Differential Revision: D4332273
fbshipit-source-id: 3d235a767963b11129c3897ad027ad761b6dae50
Summary:
Update EdenServiceHandler::getSHA1ForPath() to replace its own custom path
lookup code with EdenMount::getFileInode().
This also ended up fixing the error message to correctly include the path name
on EISDIR errors.
Reviewed By: wez
Differential Revision: D4325066
fbshipit-source-id: 9aa3932c71c33e6bc11d2c71cc8f1badb4c0dcb7
Summary:
Update the ObjectStore and BackingStore classes to have APIs that return
folly::Future objects, rather than blocking until the requested data is loaded.
For now most users still call the blocking versions of getBlob() and getTree().
Furthermore, all of the Future-based implementations actually still block
until the data is ready. I will update the code to use these new APIs in
future diffs, and then deprecate the non-future based versions.
Reviewed By: bolinfest
Differential Revision: D4318055
fbshipit-source-id: a250c23b418e69b597a4c6a95dbe80c56da5c53b
Summary:
Define InodePtr, TreeInodePtr, and FileInodePtr as aliases for std::shared_ptr
of the underlying inode type. This also updates all of the code to use these
new type names.
This will make it easier swap out std::shared_ptr with a custom pointer type in
the future. (I believe we will need a custom type in the future so that we
can have more precise control of the reference counting so we can load and
unload Inode objects on demand. std::shared_ptr::unique() doesn't quite
provide the flexibility we need, and is also being deprecated in C++17.)
Reviewed By: bolinfest
Differential Revision: D4297791
fbshipit-source-id: 1080945649290e676f62689592159f1166159b20
Summary:
In this revision, we override `committablectx.markcommitted()` to make a Thrift
call to Eden to record the new commit. For now, this defers the need for us to
implement `edendirstate.normal()`, though we expect we will have to provide a
proper implementation at some point.
Because `hg update` is not implemented yet, this puts us in a funny state where
we have to restart eden after `hg commit` to ensure all of our `TreeEntry` and
other in-memory data structures are in the correct state.
Reviewed By: simpkins
Differential Revision: D4249214
fbshipit-source-id: 8ec06dfee67070f008dd93a0ee6c810ce75d2faa
Summary:
This adds support for `hg remove <file>` in Eden and sets up some of the scaffolding
to support `hg remove <directory>`.
Note that the Thrift API for `scmRemove()` is slightly different than that of `scmAdd()`
in that it returns a list of error messages to display to the user rather than throwing
an exception. In practice, for batch operations, Mercurial will allow some operations
to succeed while others may fail, so it is possible to have multiple error messages to
return.
Unlike the current implementation of `hg add`, this does the directory traversal
on the server rather than on the client. Once we work out how to do this for
`hg remove`, we should figure out how to reuse the logic for `hg add`.
Reviewed By: simpkins
Differential Revision: D4263068
fbshipit-source-id: d084774d562c48c59664f313eba229d4197929fe
