Summary:
1.Refactored FileInode and FileData code.
2.Moved some data members of file data into struct State inside file inode
3.Refactoring code such that FileData and FileInode classes are eventually moved to FileInode class.
Reviewed By: simpkins
Differential Revision: D5414427
fbshipit-source-id: cf24721a65541ddfdec7ead4a035de4da3fd5bb5
Summary:
- Updated makeShared() to return a folly::Future<std::shared_ptr<EdenMount>> instead of just a std::shared_ptr<EdenMount>.
- Updated callers to use the returned future (get() for now)
- Refactor makeShared() and the EdenMount constructor to avoid blocking.
Reviewed By: simpkins
Differential Revision: D5424088
fbshipit-source-id: f026a3a3e4abb3593bafda76673e12c55da26322
Summary: Added header to the files that are materialized .
Reviewed By: simpkins
Differential Revision: D5387990
fbshipit-source-id: 1d551d674a39e01d6314d8bb3308e7bea3e669fc
Summary:
This updates hg_import_helper.py to override
`mercurial.txnutil._mayhavepending()` to always return True. This allows it
access to commits that are still part of a pending transaction, as long as
`writepending()` has been called on the transaction. (We could have set the
`HG_PENDING` environment variable to the repository root path instead of
monkey-patching `_mayhavepending()`, but this felt more fragile--it requires
normalizing the repository root the same way that mercurial does, and making
sure we get the correct repository root when using the share extension.)
This allows us to remove our override of `commitctx.markcommitted()` in the hg
extension. Previously we replaced this method to defer calling
`dirstate.setparents()` until the transaction was complete. We no longer have
to do this since eden can now access the pending commit.
This should also help fix bugs in the rebase workflow where we previously
failed trying to checkout a still-pending commit.
Reviewed By: wez
Differential Revision: D5345451
fbshipit-source-id: 9c32ab655254c79f030c10b7c9d02563decb04f7
Summary:
This updates the TreeInode code to remove the redundant materialized flag.
A TreeInode should have a Tree Hash if and only if it is dematerialized, so
there is no need for an extra `materialized` boolean.
This diff also fixes an issue in TreeInode::saveOverlayPostCheckout() where it
was not correctly informing it's parent TreeInode of the change if it moved
from one dematerialized state to another (with a different TreeInode hash).
This fixes the code to correctly call `parent->childDematerialized()` when it
needs to inform the parent that it now refers to a different source control
hash.
Reviewed By: wez
Differential Revision: D5336629
fbshipit-source-id: b4d86ecdef2f5faefbc243a09d869c02384ae95c
Summary:
Add a command to deserialize and display information about files in the
overlay. This can be used to help debug the current state of files in the
overlay.
Reviewed By: wez
Differential Revision: D5332014
fbshipit-source-id: 25b01579df33aa9f1926c0144e9f03aa8ece38fc
Summary:
The treemanifest import code doesn't fully work: it can end up importing the
root tree using treemanifest data, but if data for all subtrees is not also
present in the local hgcache directory it doesn't know how to load them. This
results in error messages in the log like:
failed to load inode 812: MissingKeyError: unable to find delta chain
Filesystem clients see I/O errors.
Reviewed By: wez
Differential Revision: D5382256
fbshipit-source-id: 601f6705a6382cfff80ed9c64f221b0854e5c255
Summary:
This updates the TARGETS files so that we don't run all integration tests in
both the FB and OSS build flavors any more. For the OSS build we only run the
tests in basic_tests.py now.
This reduces the overall time to run all eden tests from about 170 seconds to
about 120 seconds on my server.
Reviewed By: wez
Differential Revision: D5367406
fbshipit-source-id: 81048f415e2f69175059f346e3f81269142a5fc7
Summary:
Always call umount2() with the MNT_DETACH flag. This prevents the unmount
operation from failing if the mount point is still busy.
This is pretty much always necessary in Facebook's environment, since chg
processes tend to hold open long-lived file handles inside the mount point.
In the future we could potentially add back an option to only unmount if the
mount point is not busy, but I'll wait to do that until we find a use case for
it.
Reviewed By: akushner
Differential Revision: D5367407
fbshipit-source-id: 6805fe37346b9b35af3c40ab84c0187d8deafbd0
Summary:
Add some extra logging to the eden integration to report event timing during
tests. For now I have added some basic logging during set up and tear down.
Test setup is pretty slow for most of our tests. It looks like some of this is
spent waiting for the eden daemon to start up, but a lot of it is just waiting
on hg commands. A lot of tests currently make separate "hg add" calls for each
file they create during repository setup, and this is unfortunately really
slow. Changing these tests to make fewer "hg add" calls would probably speed
things up a fair amount.
Reviewed By: wez
Differential Revision: D5367620
fbshipit-source-id: 99270c1dc4f072b7a0c0eaed7be7730e78b9bb8c
Summary:
Update integration tests to pass logging settings in via the new `--logging`
argument instead of using `--vmodule`. Now that eden log messages have all
been switched from glog to folly logging, the `--logging` flag needs to be
used.
Reviewed By: wez
Differential Revision: D5367618
fbshipit-source-id: 299f73a8376863e46b7fffe59c539e54c78caed4
Summary:
This adds unit tests for the treemanifest import code, and fixes a couple of
bugs triggered by the tests: directory permissions were being set incorrectly,
and trees were being stored in the LocalStore with the wrong hash, causing
lookup errors later.
The HgImportTest code previously used HgImporter::importManifest(), which ended
up exercising only the flatmanifest code. This updates HgImporter to expose
separate importFlatManifest() and importTreeManifest() APIs, and updates the
tests to check both import mechanisms.
Reviewed By: wez
Differential Revision: D5365959
fbshipit-source-id: 3799ee3b937296e8025b666dd7176dbe7e268a31
Summary:
Refactor the HgImporter code to have separate functions for importing tree data
with treemanifest vs flatmanifest.
This will make it easier to unit test both code paths. In the future we may
also want to provide more control over which import mechanism to use for
particular repositories, and this will also make that easier.
Reviewed By: wez
Differential Revision: D5365957
fbshipit-source-id: 37bf7d9bc62332a1844405056cdee363dad77033
Summary:
Add == and != operators for Tree and TreeEntry.
These are mainly useful for unit tests to compare that imported trees look as
expected.
Reviewed By: wez
Differential Revision: D5365956
fbshipit-source-id: c039dfc58e430e99466db1a6c891a3c50d7906fe
Summary:
Fix HgManifestImporter to always call LocalStore::disableBatchMode()
after calling enableBatchMode(). Previously if an error occurred importing a
tree disableBatchMode() would never be called, resulting in a crash when
enableBatchMode() was called the next time we tried to import a tree.
This API is still not thread-safe, and will break if multiple callers try to
use the enableBatchMode/disableBatchMode APIs simultaneously. In the future we
should move the batch functionality to a separate class, so each caller that
wants to do batching manages their own write batch, and this isn't controlled
globally for LocalStore.
Reviewed By: wez
Differential Revision: D5365958
fbshipit-source-id: 0616f0af5029de6ebdfee768b8fddda5b6d2dfd1
Summary:
1.Added a new method to create header.
2.Added header to the overlay files of directories.
3.Added test class OverlayTest for Overlay related tests.
Reviewed By: simpkins
Differential Revision: D5335134
fbshipit-source-id: 31f59e7af70a3eeae6350261ded5d8b1bec2b9d0
Summary:
This is mostly needed because of the `folly:file` target split, but there are a few other that had changes that weren't reflected.
This also re-enables auto-deps for `multifeed/aggregator/processor/TARGETS` as it works again.
Reviewed By: yfeldblum
Differential Revision: D5362875
fbshipit-source-id: f9d2793249c0bfe644d0504f19839c108648ea3b
Summary:
[Folly] Remove `construct_in_place`.
It is an alias for `in_place`, and unnecessary.
Reviewed By: Orvid
Differential Revision: D5362354
fbshipit-source-id: 39ce07bad87185e506fe7d780789b78060008182
Summary:
Update the Overlay class to hold a lock on the info file inside the overlay for
as long as it exists.
This just adds an extra layer to help ensure that two separate edenfs processes
are not operating on the same overlay directory at the same time. We already
hold a lock on the .eden directory itself, which generally should be sufficient
protection, since overlay directories are always scoped to a .eden directory.
However, adding a per-Overlay lock will help ensure that we do not have issues
where we try to open an Overlay directory twice, particularly when remounting
an existing mount point that we just closed.
Differential Revision: D5326023
fbshipit-source-id: eb8b213225b8d6905a982db0bfac73a17d1bd246
Summary:
Add a basic unit test that adds a new file, remounts the mount point, and
confirms the materialized contents are loaded from the overlay correctly on
remount.
Differential Revision: D5326024
fbshipit-source-id: d2446030802cc4afe5af09460d590ccf8c43e525
Summary:
Add a main() function to the indoes unit test, to allow a `--logging` flag to
control log levels.
I will eventually update our default test `main()` function to include this
flag, but for now adding a custom `main()` for the inodes test is the simplest
fix.
Reviewed By: wez
Differential Revision: D5326022
fbshipit-source-id: 36f497658fdb21639408fc599cf75908b9c9acb3
Summary: It doesn't need to exist anymore
Reviewed By: yfeldblum
Differential Revision: D5318746
fbshipit-source-id: c70b184f4b3fc12ede4632d6b3d43de16ed758c7
Summary:
this code was present to help in some benchmarking
in the very early prototype code. I noticed that this was
still lingering when reviewing a diff the other day.
Remove it!
Reviewed By: simpkins
Differential Revision: D5334263
fbshipit-source-id: 9ecccf1f9922b4c8f46b2529da665e2fdf11ab7a
Summary:
Format all of the TARGETS files under eden/fs with the autodeps tool.
A few rocksdb include statements require comments so that autodeps can
correctly tell which dependency this include comes from. The rocksdb library's
source file structure unfortunately does not match the layout of how its header
files get installed, so autodeps cannot figure this out automatically.
Reviewed By: wez
Differential Revision: D5316000
fbshipit-source-id: f8163adca79ee4a673440232d6467fb83e56aa10
Summary:
Update EdenServer to catch SIGTERM and SIGINT, and to stop gracefully on these
signals.
Reviewed By: wez
Differential Revision: D5315323
fbshipit-source-id: 16611190c8d522a78cf6b624d97d4f7ecc478f96
Summary:
Upstream mercurial recently added calls to dirstate.copies().copy(),
so update the dummy_copymap() class to support this.
Reviewed By: wez
Differential Revision: D5316773
fbshipit-source-id: a2454daf265faaf2bcf1a539c8bf428f8a5e9830
Summary: Add unit tests for HgImporter to better test this import logic.
Reviewed By: bolinfest
Differential Revision: D5309171
fbshipit-source-id: b5619d8271fef1cc31a0f642ec2fcbf9eccced54
Summary: This includes adjusting them to conform to Folly's style for single file sources and headers, making targets as small as possible, not providing aggregate targets, and using auto-deps.
Reviewed By: yfeldblum
Differential Revision: D5312078
fbshipit-source-id: 1c7f5aa04da3bad236ffa23000c7bda47b82e3ef
Summary:
Update the findImportHelperPath() function to search all parent directories
containing the binary for the hg import helper script, rather than hard-coding
exactly how many directories deep we expect the edenfs binary to be placed in
the build output directory.
This makes it possible to use HgImporter in unit tests and other binaries that
aren't built in the exact same directory as the main edenfs binary.
Reviewed By: bolinfest
Differential Revision: D5309170
fbshipit-source-id: 051b26ea3017d8a9a2b9e2c59e164c28b640d1f5
Summary:
Update HgBackingStore and GitBackingStore to accept the repository path as an
AbsolutePathPiece rather than as a plain StringPiece.
Reviewed By: bolinfest
Differential Revision: D5309172
fbshipit-source-id: aca4818c3add11d07ece796298f6175ca4fb1448
Summary:
1. Moved read, write, mkdir, rm methods in hg/lib/hg_extension_test_base.py to lib/test_case.py.
2. Added integration test case to test unload free inodes.
Reviewed By: simpkins
Differential Revision: D5277870
fbshipit-source-id: b93b6049a10357cf8c92366e6dca3968f7f30c30
Summary:
Update eden to log via the new folly logging APIs rather than with glog.
This adds a new --logging flag that takes a logging configuration string.
By default we set the log level to INFO for all eden logs, and WARNING for
everything else. (I suspect we may eventually want to run with some
high-priority debug logs enabled for some or all of eden, but this seems like a
reasonable default to start with.)
Reviewed By: wez
Differential Revision: D5290783
fbshipit-source-id: 14183489c48c96613e2aca0f513bfa82fd9798c7
Summary:
Now that the non-future versions of these APIs have been removed, rename
getBlobFuture() to getBlob(), and getTreeFuture() to getTree()
Reviewed By: wez
Differential Revision: D5295690
fbshipit-source-id: 30dcb88854b23160692b9cd83a632f863e07b491
Summary:
Remove the blocking getBlob() API.
There were a few call sites in FileData still using this blocking API. For now
I simply updated them to use getBlobFuture() and make a blocking get() call on
the returned future. These call sites already had TODO comments documenting
the blocking behavior.
I plan to rename getBlobFuture() to getBlob() in a subsequent diff.
Reviewed By: wez
Differential Revision: D5295726
fbshipit-source-id: 748fd7a140b9b59da339a330071f732bba38cb35
Summary:
Remove the blocking getTree() API. All call sites are using getTreeFuture()
instead now.
I plan to rename getTreeFuture() to getTree() in a subsequent diff.
Reviewed By: wez
Differential Revision: D5295725
fbshipit-source-id: 6b40b4c808da94a9c68decae3ce38c7d13fbe9f5
Summary:
Remove the ObjectStores.h and .cpp files. These files contained helper
functions for looking up Tree and TreeEntry objects based on a deep relative
path.
These functions are no longer used anywhere anymore (the code that performs
deep lookups always does inode lookups instead). Additionally, this file is
the only place still using some of the blocking, non-Future-based APIs for
object lookups. Deleting these files will let us remove these deprecated
blocking APIs.
Reviewed By: wez
Differential Revision: D5295691
fbshipit-source-id: 89229827305490eba4d3a581954a90c5cf63238d
Summary:
1.Added a new thrift method to unload free inodes of a directory and its sub directories.
2.Added a new debug sub command 'eden debug unload <path>' to cli tools to unload free inodes.
Reviewed By: simpkins
Differential Revision: D5261038
fbshipit-source-id: 85b4c5ae18c0ae24c666a44ac9892765e753397f
Summary:
1.Added new rage command to command line tools.
2.Eden rage command currently shows Logs,package version,rpm version,build revision,build upstream revision,list of running eden processes and their info,list of mount points and their info.
Reviewed By: simpkins
Differential Revision: D5220250
fbshipit-source-id: 357f46d8d08d4a1f197b705dfd1a28668dd180f0
Summary: made the relative path return "" when mount point directory is returned "."
Reviewed By: simpkins
Differential Revision: D5250543
fbshipit-source-id: bd0d648db8030cfc6ca952fccfb9972f18290b5e
Summary:
Depend on D4492997
Currently, we assume that changes applied to HwSwitch always succeed (or the process exits). There is a case where we want to keep moving despite failures: when we get excessive number of route (probably) because of some misconfiguration. This diff is a step towards taking care of such situation.
We change the interface between SwSwitch and HwSwitch so that HwSwitch, when asked to apply delta of old and new state, returns either the new state if it succeeds, or returns a "pruned version of new state" if some elements of new state it was not able to apply.
SwSwitch now keeps track of two states, a state that has been applied in the hardware, and another one that is desired in hardware. SwSwitch strives to catch applied state to the desired state (while desired state is also changes as move state updates are arriving).
One issue with this diff: Warmboot is not preserving "unapplied" routes, i.e., unapplied routes get lost across warmboots. This will be fixed in a seperate diff.
One naming issue: I thought about keeping SwSwitch::getState() name intact, but thought that it is not a good idea to keep distinction of states opaque to callers. So, decided to change the name.
Reviewed By: ninas
Differential Revision: D4405751
fbshipit-source-id: c933b4418445ae49d5b8e21dea95e437a8e1593d
Summary:
This is a major change to Eden's Hg extension.
Our initial attempt to implement `edendirstate` was to create a "clean room"
implementation that did not share code with `mercurial/dirstate.py`. This was
helpful in uncovering the subset of the dirstate API that matters for Eden. It
also provided a better safeguard against upstream changes to `dirstate.py` in
Mercurial itself.
In this implementation, the state transition management was mostly done
on the server in `Dirstate.cpp`. We also made a modest attempt to make
`Dirstate.cpp` "SCM-agnostic" such that the same APIs could be used for
Git at some point.
However, as we have tried to support more of the sophisticated functionality
in Mercurial, particularly `hg histedit`, achieving parity between the clean room
implementation and Mercurial's internals has become more challenging.
Ultimately, the clean room implementation is likely the right way to go for Eden,
but for now, we need to prioritize having feature parity with vanilla Hg when
using Eden. Once we have a more complete set of integration tests in place,
we can reimplement Eden's dirstate more aggressively to optimize things.
Fortunately, the [[ https://bitbucket.org/facebook/hg-experimental/src/default/sqldirstate/ | sqldirstate ]]
extension has already demonstrated that it is possible to provide a faithful
dirstate implementation that subclasses the original `dirstate` while using a different
storage mechanism. As such, I used `sqldirstate` as a model when implementing
the new `eden_dirstate` (distinguishing it from our v1 implementation, `edendirstate`).
In particular, `sqldirstate` uses SQL tables as storage for the following private fields
of `dirstate`: `_map`, `_dirs`, `_copymap`, `_filefoldmap`, `_dirfoldmap`. Because
`_filefoldmap` and `_dirfoldmap` exist to deal with case-insensitivity issues, we
do not support them in `eden_dirstate` and add code to ensure the codepaths that
would access them in `dirstate` never get exercised. Similarly, we also implemented
`eden_dirstate` so that it never accesses `_dirs`. (`_dirs` is a multiset of all directories in the
dirstate, which is an O(repo) data structure, so we do not want to maintain it in Eden.
It appears to be primarily used for checking whether a path to a file already exists in
the dirstate as a directory. We can protect against that in more efficient ways.)
That leaves only `_map` and `_copymap` to worry about. `_copymap` contains the set
of files that have been marked "copied" in the current dirstate, so it is fairly small and
can be stored on disk or in memory with little concern. `_map` is a bit trickier because
it is expected to have an entry for every file in the dirstate. In `sqldirstate`, it is stored
across two tables: `files` and `nonnormalfiles`. For Eden, we already represent the data
analogous to the `files` table in RocksDB/the overlay, so we do not need to create a new
equivalent to the `files` table. We do, however, need an equivalent to the `nonnormalfiles`
table, which we store in as Thrift-serialized data in an ordinary file along with the `_copymap`
data.
In our Hg extension, our implementation of `_map` is `eden_dirstate_map`, which is defined
in a Python file of the same name. Our implementation of `_copymap` is `dummy_copymap`,
which is defined in `eden_dirstate.py`. Both of these collections are simple pass-through data
structures that translate their method calls to Thrift server calls. I expect we will want to
optimize this in the future via some client-side caching, as well as creating batch APIs for talking
to the server via Thrift.
One advantage of this new implementation is that it enables us to delete
`eden/hg/eden/overrides.py`, which overrode the entry points for `hg add` and `hg remove`.
Between the recent implementation of `dirstate.walk()` for Eden and this switch
to the real dirstate, we can now use the default implementation of `hg add` and `hg remove`
(although we have to play some tricks, like in the implementation of `eden_dirstate.status()`
in order to make `hg remove` work).
In the course of doing this revision, I discovered that I had to make a minor fix to
`EdenMatchInfo.make_glob_list()` because `hg add foo` was being treated as
`hg add foo/**/*` even when `foo` was just a file (as opposed to a directory), in which
case the glob was not matching `foo`!
I also had to do some work in `eden_dirstate.status()` in which the `match` argument
was previously largely ignored. It turns out that `dirstate.py` uses `status()` for a number
of things with the `match` specified as a filter, so the output of `status()` must be filtered
by `match` accordingly. Ultimately, this seems like work that would be better done on the
server, but for simplicity, we're just going to do it in Python, for now.
For the reasons explained above, this revision deletes a lot of code `Dirstate.cpp`.
As such, `DirstateTest.cpp` does not seem worth refactoring, though the scenarios it was
testing should probably be converted to integration tests. At a high level, the role of
`DirstatePersistence` has not changed, but the exact data it writes is much different.
Its corresponding unit test is also disabled, for now.
Note that this revision does not change the name of the file where "dirstate data" is written
(this is defined as `kDirstateFile` in `ClientConfig.cpp`), so we should blow away any existing
instances of this file once this change lands. (It is still early enough in the project that it does
not seem worth the overhead of a proper migration.)
The true test of the success of this new approach is the ease with which we can write more
integration tests for things like `hg histedit` and `hg graft`. Ideally, these should require very
few changes to `eden_dirstate.py`.
Reviewed By: simpkins
Differential Revision: D5071778
fbshipit-source-id: e8fec4d393035d80f36516ac050cad025dc3ba31
Summary:
This is generated by applying clang-tidy `-checks=modernize-use-override` to all the c++ code in project eden.
It enforces the use of the keywords `virtual`, `verride` and `final` in a way compliant to the style guide.
Reviewed By: igorsugak
Differential Revision: D5108807
fbshipit-source-id: 596f2d73f1137de350114416edb1c37da5423ed5
Summary: The `nodiscard` attribute was added to standard C++ in C++17. Prefer the standard formulation when it is available; otherwise, use `__attribute__((__warn_unused_result__))` on compilers that support the GNU extensions, and `_Check_return_` on MSVC. The old `FOLLY_WARN_UNUSED_RESULT` is now expands to `FOLLY_NODISCARD`.
Reviewed By: yfeldblum
Differential Revision: D5105137
fbshipit-source-id: 9aa22e81cd9f0b89f9343433aeae3ba365227ccb
Summary:
I was working on a new test and I got an error that `directaccess` must be
enabled for `inhibit` to work.
Reviewed By: simpkins
Differential Revision: D5077133
fbshipit-source-id: cc5235c845e3f299f96e1c901ef4aea18ca57b76
Summary:
We're using `window` to constrain resources and maintain an upper
bound on the number of futures that we spawn. Ironically, for a largish number
of already completed futures, this results in a recursive chain of `spawn`
calls in the `window` implementation and results in a stack overflow (the kind
that crashes a program, not the kind that answers programming questions).
This diff removes the use of `window` and replaces it with `collect`.
An earlier iteration of this diff just made all of these calls serially,
blocking the main thread. This at least does things in parallel; both
can torture the system for pathological walks, but at least the parallel
one should complete faster.
Reviewed By: simpkins
Differential Revision: D4932774
fbshipit-source-id: 40d39a85029f38ff69a530070efb879a81950451
