2017-06-28 07:08:54 +03:00
|
|
|
# @autodeps
|
|
|
|
|
|
2016-09-10 02:56:02 +03:00
|
|
|
thrift_library(
|
2017-06-02 03:41:11 +03:00
|
|
|
name = "serialization",
|
2017-07-08 04:32:42 +03:00
|
|
|
languages = [
|
|
|
|
|
"cpp2",
|
|
|
|
|
"py",
|
|
|
|
|
],
|
|
|
|
|
py_base_module = "facebook.eden",
|
2017-06-02 03:41:11 +03:00
|
|
|
thrift_args = ["--strict"],
|
|
|
|
|
thrift_srcs = {
|
|
|
|
|
"overlay.thrift": [],
|
|
|
|
|
},
|
|
|
|
|
deps = [
|
|
|
|
|
":hgdirstate",
|
|
|
|
|
],
|
Reimplement dirstate used by Eden's Hg extension as a subclass of Hg's dirstate.
Summary:
This is a major change to Eden's Hg extension.
Our initial attempt to implement `edendirstate` was to create a "clean room"
implementation that did not share code with `mercurial/dirstate.py`. This was
helpful in uncovering the subset of the dirstate API that matters for Eden. It
also provided a better safeguard against upstream changes to `dirstate.py` in
Mercurial itself.
In this implementation, the state transition management was mostly done
on the server in `Dirstate.cpp`. We also made a modest attempt to make
`Dirstate.cpp` "SCM-agnostic" such that the same APIs could be used for
Git at some point.
However, as we have tried to support more of the sophisticated functionality
in Mercurial, particularly `hg histedit`, achieving parity between the clean room
implementation and Mercurial's internals has become more challenging.
Ultimately, the clean room implementation is likely the right way to go for Eden,
but for now, we need to prioritize having feature parity with vanilla Hg when
using Eden. Once we have a more complete set of integration tests in place,
we can reimplement Eden's dirstate more aggressively to optimize things.
Fortunately, the [[ https://bitbucket.org/facebook/hg-experimental/src/default/sqldirstate/ | sqldirstate ]]
extension has already demonstrated that it is possible to provide a faithful
dirstate implementation that subclasses the original `dirstate` while using a different
storage mechanism. As such, I used `sqldirstate` as a model when implementing
the new `eden_dirstate` (distinguishing it from our v1 implementation, `edendirstate`).
In particular, `sqldirstate` uses SQL tables as storage for the following private fields
of `dirstate`: `_map`, `_dirs`, `_copymap`, `_filefoldmap`, `_dirfoldmap`. Because
`_filefoldmap` and `_dirfoldmap` exist to deal with case-insensitivity issues, we
do not support them in `eden_dirstate` and add code to ensure the codepaths that
would access them in `dirstate` never get exercised. Similarly, we also implemented
`eden_dirstate` so that it never accesses `_dirs`. (`_dirs` is a multiset of all directories in the
dirstate, which is an O(repo) data structure, so we do not want to maintain it in Eden.
It appears to be primarily used for checking whether a path to a file already exists in
the dirstate as a directory. We can protect against that in more efficient ways.)
That leaves only `_map` and `_copymap` to worry about. `_copymap` contains the set
of files that have been marked "copied" in the current dirstate, so it is fairly small and
can be stored on disk or in memory with little concern. `_map` is a bit trickier because
it is expected to have an entry for every file in the dirstate. In `sqldirstate`, it is stored
across two tables: `files` and `nonnormalfiles`. For Eden, we already represent the data
analogous to the `files` table in RocksDB/the overlay, so we do not need to create a new
equivalent to the `files` table. We do, however, need an equivalent to the `nonnormalfiles`
table, which we store in as Thrift-serialized data in an ordinary file along with the `_copymap`
data.
In our Hg extension, our implementation of `_map` is `eden_dirstate_map`, which is defined
in a Python file of the same name. Our implementation of `_copymap` is `dummy_copymap`,
which is defined in `eden_dirstate.py`. Both of these collections are simple pass-through data
structures that translate their method calls to Thrift server calls. I expect we will want to
optimize this in the future via some client-side caching, as well as creating batch APIs for talking
to the server via Thrift.
One advantage of this new implementation is that it enables us to delete
`eden/hg/eden/overrides.py`, which overrode the entry points for `hg add` and `hg remove`.
Between the recent implementation of `dirstate.walk()` for Eden and this switch
to the real dirstate, we can now use the default implementation of `hg add` and `hg remove`
(although we have to play some tricks, like in the implementation of `eden_dirstate.status()`
in order to make `hg remove` work).
In the course of doing this revision, I discovered that I had to make a minor fix to
`EdenMatchInfo.make_glob_list()` because `hg add foo` was being treated as
`hg add foo/**/*` even when `foo` was just a file (as opposed to a directory), in which
case the glob was not matching `foo`!
I also had to do some work in `eden_dirstate.status()` in which the `match` argument
was previously largely ignored. It turns out that `dirstate.py` uses `status()` for a number
of things with the `match` specified as a filter, so the output of `status()` must be filtered
by `match` accordingly. Ultimately, this seems like work that would be better done on the
server, but for simplicity, we're just going to do it in Python, for now.
For the reasons explained above, this revision deletes a lot of code `Dirstate.cpp`.
As such, `DirstateTest.cpp` does not seem worth refactoring, though the scenarios it was
testing should probably be converted to integration tests. At a high level, the role of
`DirstatePersistence` has not changed, but the exact data it writes is much different.
Its corresponding unit test is also disabled, for now.
Note that this revision does not change the name of the file where "dirstate data" is written
(this is defined as `kDirstateFile` in `ClientConfig.cpp`), so we should blow away any existing
instances of this file once this change lands. (It is still early enough in the project that it does
not seem worth the overhead of a proper migration.)
The true test of the success of this new approach is the ease with which we can write more
integration tests for things like `hg histedit` and `hg graft`. Ideally, these should require very
few changes to `eden_dirstate.py`.
Reviewed By: simpkins
Differential Revision: D5071778
fbshipit-source-id: e8fec4d393035d80f36516ac050cad025dc3ba31
2017-05-26 21:51:30 +03:00
|
|
|
)
|
|
|
|
|
|
|
|
|
|
thrift_library(
|
2017-06-02 03:41:11 +03:00
|
|
|
name = "hgdirstate",
|
|
|
|
|
# Need to support .java and .py because eden.thrift depends on this file.
|
|
|
|
|
languages = [
|
|
|
|
|
"cpp2",
|
|
|
|
|
"java",
|
|
|
|
|
"py",
|
|
|
|
|
],
|
|
|
|
|
py_base_module = "facebook",
|
|
|
|
|
thrift_args = ["--strict"],
|
|
|
|
|
thrift_srcs = {
|
|
|
|
|
"hgdirstate.thrift": None,
|
|
|
|
|
},
|
2016-09-10 02:56:02 +03:00
|
|
|
)
|
|
|
|
|
|
2016-05-13 02:07:33 +03:00
|
|
|
cpp_library(
|
2017-06-02 03:41:11 +03:00
|
|
|
name = "inodes",
|
|
|
|
|
srcs = glob(["*.cpp"]),
|
|
|
|
|
headers = glob(["*.h"]),
|
|
|
|
|
deps = [
|
|
|
|
|
":hgdirstate-cpp2",
|
|
|
|
|
":serialization-cpp2",
|
|
|
|
|
"@/eden/fs/config:config",
|
|
|
|
|
"@/eden/fs/fuse:fusell",
|
2017-09-09 05:15:20 +03:00
|
|
|
"@/eden/fs/fuse/privhelper:privhelper",
|
2017-06-02 03:41:11 +03:00
|
|
|
"@/eden/fs/journal:journal",
|
|
|
|
|
"@/eden/fs/model:model",
|
|
|
|
|
"@/eden/fs/model/git:gitignore",
|
2017-06-28 07:08:54 +03:00
|
|
|
"@/eden/fs/service:thrift-streaming-cpp2",
|
2017-06-02 03:41:11 +03:00
|
|
|
"@/eden/fs/service:thrift_cpp",
|
|
|
|
|
"@/eden/fs/store:store",
|
fix EdenServer::unmount() to fully wait for mount point cleanup
Summary:
This fixes EdenServer::unmount() to actually wait for all EdenMount cleanup
to complete, and fixes unmountAll() to return a Future that correctly waits for
all mount points to be cleaned up.
Previously `unmount()` waited for the mount point to be unmounted from the
kernel, but did not wait for EdenMount shutdown to complete. Previously
EdenMount shutdown was not triggered until the last reference to the
shared_ptr<EdenMount> was released. This often happened in the FUSE channel
thread that triggered the mountFinished() call--it would still hold a
reference to this pointer, and would not release it until after
mountFinished() returns. As a result, when the main thread was shutting down,
`main()` would call `unmountAll()`, and then return soon after it completed.
Some FUSE channel threads may still be running at this point, still performing
`EdenMount` shutdown while the main thread was exiting. This could result in
crashes and deadlocks as shutdown tried to access objects already destroyed by
the main thread.
With this change `EdenMount::shutdown()` is triggered explicitly during
`mountFinished()`, and `unmount()` will not complete until this finishes.
The `EdenMount` object may still exist at this point, and could still be
deleted by the FUSE channel thread, but the deletion now only requires freeing
the memory and does not require accessing other data that may have been cleaned
up by the main thread.
We should still clean up the FUSE channel thread handling in the future, to
make sure these threads are joined before the main thread exits. However, that
cleanup can wait until a separate diff. Ideally I would like to move more of
the mount and unmount logic from EdenServer and EdenServiceHandler and put that
code in EdenMount instead.
Reviewed By: bolinfest
Differential Revision: D5541318
fbshipit-source-id: 470332478357a85c314bc40458373cb0f827f62b
2017-08-03 02:52:18 +03:00
|
|
|
"@/eden/fs/utils:path",
|
2017-06-02 03:41:11 +03:00
|
|
|
"@/eden/fs/utils:utils",
|
2017-06-28 07:08:54 +03:00
|
|
|
"@/folly:exception",
|
|
|
|
|
"@/folly:exception_wrapper",
|
fix EdenServer::unmount() to fully wait for mount point cleanup
Summary:
This fixes EdenServer::unmount() to actually wait for all EdenMount cleanup
to complete, and fixes unmountAll() to return a Future that correctly waits for
all mount points to be cleaned up.
Previously `unmount()` waited for the mount point to be unmounted from the
kernel, but did not wait for EdenMount shutdown to complete. Previously
EdenMount shutdown was not triggered until the last reference to the
shared_ptr<EdenMount> was released. This often happened in the FUSE channel
thread that triggered the mountFinished() call--it would still hold a
reference to this pointer, and would not release it until after
mountFinished() returns. As a result, when the main thread was shutting down,
`main()` would call `unmountAll()`, and then return soon after it completed.
Some FUSE channel threads may still be running at this point, still performing
`EdenMount` shutdown while the main thread was exiting. This could result in
crashes and deadlocks as shutdown tried to access objects already destroyed by
the main thread.
With this change `EdenMount::shutdown()` is triggered explicitly during
`mountFinished()`, and `unmount()` will not complete until this finishes.
The `EdenMount` object may still exist at this point, and could still be
deleted by the FUSE channel thread, but the deletion now only requires freeing
the memory and does not require accessing other data that may have been cleaned
up by the main thread.
We should still clean up the FUSE channel thread handling in the future, to
make sure these threads are joined before the main thread exits. However, that
cleanup can wait until a separate diff. Ideally I would like to move more of
the mount and unmount logic from EdenServer and EdenServiceHandler and put that
code in EdenMount instead.
Reviewed By: bolinfest
Differential Revision: D5541318
fbshipit-source-id: 470332478357a85c314bc40458373cb0f827f62b
2017-08-03 02:52:18 +03:00
|
|
|
"@/folly:fbstring",
|
2017-06-28 07:08:54 +03:00
|
|
|
"@/folly:file",
|
2017-07-03 01:43:46 +03:00
|
|
|
"@/folly:file_util",
|
2017-06-28 07:08:54 +03:00
|
|
|
"@/folly:format",
|
|
|
|
|
"@/folly:likely",
|
|
|
|
|
"@/folly:map_util",
|
|
|
|
|
"@/folly:optional",
|
|
|
|
|
"@/folly:portability",
|
|
|
|
|
"@/folly:range",
|
|
|
|
|
"@/folly:shared_mutex",
|
2017-06-30 06:13:49 +03:00
|
|
|
"@/folly:string",
|
2017-06-28 07:08:54 +03:00
|
|
|
"@/folly:synchronized",
|
|
|
|
|
"@/folly:thread_local",
|
2017-09-09 05:15:20 +03:00
|
|
|
"@/folly:thread_name",
|
2017-06-28 07:08:54 +03:00
|
|
|
"@/folly:unit",
|
|
|
|
|
"@/folly/experimental:string_keyed_unordered_map",
|
2017-06-22 23:39:57 +03:00
|
|
|
"@/folly/experimental/logging:logging",
|
2017-06-28 07:08:54 +03:00
|
|
|
"@/folly/futures:core",
|
|
|
|
|
"@/folly/io:iobuf",
|
|
|
|
|
"@/thrift/lib/cpp2/protocol:protocol",
|
|
|
|
|
"@/wangle:wangle",
|
2017-06-02 03:41:11 +03:00
|
|
|
],
|
|
|
|
|
external_deps = [
|
2017-06-28 07:08:54 +03:00
|
|
|
"boost",
|
2017-06-02 03:41:11 +03:00
|
|
|
"gflags",
|
2017-06-28 07:08:54 +03:00
|
|
|
"glog",
|
|
|
|
|
("boost", None, "boost_filesystem"),
|
|
|
|
|
("openssl", None, "crypto"),
|
2017-06-02 03:41:11 +03:00
|
|
|
],
|
2016-05-13 02:07:33 +03:00
|
|
|
)
|
