Summary:
This is mostly needed because of the `folly:file` target split, but there are a few other that had changes that weren't reflected.
This also re-enables auto-deps for `multifeed/aggregator/processor/TARGETS` as it works again.
Reviewed By: yfeldblum
Differential Revision: D5362875
fbshipit-source-id: f9d2793249c0bfe644d0504f19839c108648ea3b
Summary:
[Folly] Remove `construct_in_place`.
It is an alias for `in_place`, and unnecessary.
Reviewed By: Orvid
Differential Revision: D5362354
fbshipit-source-id: 39ce07bad87185e506fe7d780789b78060008182
Summary:
Update the Overlay class to hold a lock on the info file inside the overlay for
as long as it exists.
This just adds an extra layer to help ensure that two separate edenfs processes
are not operating on the same overlay directory at the same time. We already
hold a lock on the .eden directory itself, which generally should be sufficient
protection, since overlay directories are always scoped to a .eden directory.
However, adding a per-Overlay lock will help ensure that we do not have issues
where we try to open an Overlay directory twice, particularly when remounting
an existing mount point that we just closed.
Differential Revision: D5326023
fbshipit-source-id: eb8b213225b8d6905a982db0bfac73a17d1bd246
Summary:
Add a basic unit test that adds a new file, remounts the mount point, and
confirms the materialized contents are loaded from the overlay correctly on
remount.
Differential Revision: D5326024
fbshipit-source-id: d2446030802cc4afe5af09460d590ccf8c43e525
Summary:
Add a main() function to the indoes unit test, to allow a `--logging` flag to
control log levels.
I will eventually update our default test `main()` function to include this
flag, but for now adding a custom `main()` for the inodes test is the simplest
fix.
Reviewed By: wez
Differential Revision: D5326022
fbshipit-source-id: 36f497658fdb21639408fc599cf75908b9c9acb3
Summary: It doesn't need to exist anymore
Reviewed By: yfeldblum
Differential Revision: D5318746
fbshipit-source-id: c70b184f4b3fc12ede4632d6b3d43de16ed758c7
Summary:
this code was present to help in some benchmarking
in the very early prototype code. I noticed that this was
still lingering when reviewing a diff the other day.
Remove it!
Reviewed By: simpkins
Differential Revision: D5334263
fbshipit-source-id: 9ecccf1f9922b4c8f46b2529da665e2fdf11ab7a
Summary:
Format all of the TARGETS files under eden/fs with the autodeps tool.
A few rocksdb include statements require comments so that autodeps can
correctly tell which dependency this include comes from. The rocksdb library's
source file structure unfortunately does not match the layout of how its header
files get installed, so autodeps cannot figure this out automatically.
Reviewed By: wez
Differential Revision: D5316000
fbshipit-source-id: f8163adca79ee4a673440232d6467fb83e56aa10
Summary:
Update EdenServer to catch SIGTERM and SIGINT, and to stop gracefully on these
signals.
Reviewed By: wez
Differential Revision: D5315323
fbshipit-source-id: 16611190c8d522a78cf6b624d97d4f7ecc478f96
Summary:
Upstream mercurial recently added calls to dirstate.copies().copy(),
so update the dummy_copymap() class to support this.
Reviewed By: wez
Differential Revision: D5316773
fbshipit-source-id: a2454daf265faaf2bcf1a539c8bf428f8a5e9830
Summary: Add unit tests for HgImporter to better test this import logic.
Reviewed By: bolinfest
Differential Revision: D5309171
fbshipit-source-id: b5619d8271fef1cc31a0f642ec2fcbf9eccced54
Summary: This includes adjusting them to conform to Folly's style for single file sources and headers, making targets as small as possible, not providing aggregate targets, and using auto-deps.
Reviewed By: yfeldblum
Differential Revision: D5312078
fbshipit-source-id: 1c7f5aa04da3bad236ffa23000c7bda47b82e3ef
Summary:
Update the findImportHelperPath() function to search all parent directories
containing the binary for the hg import helper script, rather than hard-coding
exactly how many directories deep we expect the edenfs binary to be placed in
the build output directory.
This makes it possible to use HgImporter in unit tests and other binaries that
aren't built in the exact same directory as the main edenfs binary.
Reviewed By: bolinfest
Differential Revision: D5309170
fbshipit-source-id: 051b26ea3017d8a9a2b9e2c59e164c28b640d1f5
Summary:
Update HgBackingStore and GitBackingStore to accept the repository path as an
AbsolutePathPiece rather than as a plain StringPiece.
Reviewed By: bolinfest
Differential Revision: D5309172
fbshipit-source-id: aca4818c3add11d07ece796298f6175ca4fb1448
Summary:
1. Moved read, write, mkdir, rm methods in hg/lib/hg_extension_test_base.py to lib/test_case.py.
2. Added integration test case to test unload free inodes.
Reviewed By: simpkins
Differential Revision: D5277870
fbshipit-source-id: b93b6049a10357cf8c92366e6dca3968f7f30c30
Summary:
Update eden to log via the new folly logging APIs rather than with glog.
This adds a new --logging flag that takes a logging configuration string.
By default we set the log level to INFO for all eden logs, and WARNING for
everything else. (I suspect we may eventually want to run with some
high-priority debug logs enabled for some or all of eden, but this seems like a
reasonable default to start with.)
Reviewed By: wez
Differential Revision: D5290783
fbshipit-source-id: 14183489c48c96613e2aca0f513bfa82fd9798c7
Summary:
Now that the non-future versions of these APIs have been removed, rename
getBlobFuture() to getBlob(), and getTreeFuture() to getTree()
Reviewed By: wez
Differential Revision: D5295690
fbshipit-source-id: 30dcb88854b23160692b9cd83a632f863e07b491
Summary:
Remove the blocking getBlob() API.
There were a few call sites in FileData still using this blocking API. For now
I simply updated them to use getBlobFuture() and make a blocking get() call on
the returned future. These call sites already had TODO comments documenting
the blocking behavior.
I plan to rename getBlobFuture() to getBlob() in a subsequent diff.
Reviewed By: wez
Differential Revision: D5295726
fbshipit-source-id: 748fd7a140b9b59da339a330071f732bba38cb35
Summary:
Remove the blocking getTree() API. All call sites are using getTreeFuture()
instead now.
I plan to rename getTreeFuture() to getTree() in a subsequent diff.
Reviewed By: wez
Differential Revision: D5295725
fbshipit-source-id: 6b40b4c808da94a9c68decae3ce38c7d13fbe9f5
Summary:
Remove the ObjectStores.h and .cpp files. These files contained helper
functions for looking up Tree and TreeEntry objects based on a deep relative
path.
These functions are no longer used anywhere anymore (the code that performs
deep lookups always does inode lookups instead). Additionally, this file is
the only place still using some of the blocking, non-Future-based APIs for
object lookups. Deleting these files will let us remove these deprecated
blocking APIs.
Reviewed By: wez
Differential Revision: D5295691
fbshipit-source-id: 89229827305490eba4d3a581954a90c5cf63238d
Summary:
1.Added a new thrift method to unload free inodes of a directory and its sub directories.
2.Added a new debug sub command 'eden debug unload <path>' to cli tools to unload free inodes.
Reviewed By: simpkins
Differential Revision: D5261038
fbshipit-source-id: 85b4c5ae18c0ae24c666a44ac9892765e753397f
Summary:
1.Added new rage command to command line tools.
2.Eden rage command currently shows Logs,package version,rpm version,build revision,build upstream revision,list of running eden processes and their info,list of mount points and their info.
Reviewed By: simpkins
Differential Revision: D5220250
fbshipit-source-id: 357f46d8d08d4a1f197b705dfd1a28668dd180f0
Summary: made the relative path return "" when mount point directory is returned "."
Reviewed By: simpkins
Differential Revision: D5250543
fbshipit-source-id: bd0d648db8030cfc6ca952fccfb9972f18290b5e
Summary:
Depend on D4492997
Currently, we assume that changes applied to HwSwitch always succeed (or the process exits). There is a case where we want to keep moving despite failures: when we get excessive number of route (probably) because of some misconfiguration. This diff is a step towards taking care of such situation.
We change the interface between SwSwitch and HwSwitch so that HwSwitch, when asked to apply delta of old and new state, returns either the new state if it succeeds, or returns a "pruned version of new state" if some elements of new state it was not able to apply.
SwSwitch now keeps track of two states, a state that has been applied in the hardware, and another one that is desired in hardware. SwSwitch strives to catch applied state to the desired state (while desired state is also changes as move state updates are arriving).
One issue with this diff: Warmboot is not preserving "unapplied" routes, i.e., unapplied routes get lost across warmboots. This will be fixed in a seperate diff.
One naming issue: I thought about keeping SwSwitch::getState() name intact, but thought that it is not a good idea to keep distinction of states opaque to callers. So, decided to change the name.
Reviewed By: ninas
Differential Revision: D4405751
fbshipit-source-id: c933b4418445ae49d5b8e21dea95e437a8e1593d
Summary:
This is a major change to Eden's Hg extension.
Our initial attempt to implement `edendirstate` was to create a "clean room"
implementation that did not share code with `mercurial/dirstate.py`. This was
helpful in uncovering the subset of the dirstate API that matters for Eden. It
also provided a better safeguard against upstream changes to `dirstate.py` in
Mercurial itself.
In this implementation, the state transition management was mostly done
on the server in `Dirstate.cpp`. We also made a modest attempt to make
`Dirstate.cpp` "SCM-agnostic" such that the same APIs could be used for
Git at some point.
However, as we have tried to support more of the sophisticated functionality
in Mercurial, particularly `hg histedit`, achieving parity between the clean room
implementation and Mercurial's internals has become more challenging.
Ultimately, the clean room implementation is likely the right way to go for Eden,
but for now, we need to prioritize having feature parity with vanilla Hg when
using Eden. Once we have a more complete set of integration tests in place,
we can reimplement Eden's dirstate more aggressively to optimize things.
Fortunately, the [[ https://bitbucket.org/facebook/hg-experimental/src/default/sqldirstate/ | sqldirstate ]]
extension has already demonstrated that it is possible to provide a faithful
dirstate implementation that subclasses the original `dirstate` while using a different
storage mechanism. As such, I used `sqldirstate` as a model when implementing
the new `eden_dirstate` (distinguishing it from our v1 implementation, `edendirstate`).
In particular, `sqldirstate` uses SQL tables as storage for the following private fields
of `dirstate`: `_map`, `_dirs`, `_copymap`, `_filefoldmap`, `_dirfoldmap`. Because
`_filefoldmap` and `_dirfoldmap` exist to deal with case-insensitivity issues, we
do not support them in `eden_dirstate` and add code to ensure the codepaths that
would access them in `dirstate` never get exercised. Similarly, we also implemented
`eden_dirstate` so that it never accesses `_dirs`. (`_dirs` is a multiset of all directories in the
dirstate, which is an O(repo) data structure, so we do not want to maintain it in Eden.
It appears to be primarily used for checking whether a path to a file already exists in
the dirstate as a directory. We can protect against that in more efficient ways.)
That leaves only `_map` and `_copymap` to worry about. `_copymap` contains the set
of files that have been marked "copied" in the current dirstate, so it is fairly small and
can be stored on disk or in memory with little concern. `_map` is a bit trickier because
it is expected to have an entry for every file in the dirstate. In `sqldirstate`, it is stored
across two tables: `files` and `nonnormalfiles`. For Eden, we already represent the data
analogous to the `files` table in RocksDB/the overlay, so we do not need to create a new
equivalent to the `files` table. We do, however, need an equivalent to the `nonnormalfiles`
table, which we store in as Thrift-serialized data in an ordinary file along with the `_copymap`
data.
In our Hg extension, our implementation of `_map` is `eden_dirstate_map`, which is defined
in a Python file of the same name. Our implementation of `_copymap` is `dummy_copymap`,
which is defined in `eden_dirstate.py`. Both of these collections are simple pass-through data
structures that translate their method calls to Thrift server calls. I expect we will want to
optimize this in the future via some client-side caching, as well as creating batch APIs for talking
to the server via Thrift.
One advantage of this new implementation is that it enables us to delete
`eden/hg/eden/overrides.py`, which overrode the entry points for `hg add` and `hg remove`.
Between the recent implementation of `dirstate.walk()` for Eden and this switch
to the real dirstate, we can now use the default implementation of `hg add` and `hg remove`
(although we have to play some tricks, like in the implementation of `eden_dirstate.status()`
in order to make `hg remove` work).
In the course of doing this revision, I discovered that I had to make a minor fix to
`EdenMatchInfo.make_glob_list()` because `hg add foo` was being treated as
`hg add foo/**/*` even when `foo` was just a file (as opposed to a directory), in which
case the glob was not matching `foo`!
I also had to do some work in `eden_dirstate.status()` in which the `match` argument
was previously largely ignored. It turns out that `dirstate.py` uses `status()` for a number
of things with the `match` specified as a filter, so the output of `status()` must be filtered
by `match` accordingly. Ultimately, this seems like work that would be better done on the
server, but for simplicity, we're just going to do it in Python, for now.
For the reasons explained above, this revision deletes a lot of code `Dirstate.cpp`.
As such, `DirstateTest.cpp` does not seem worth refactoring, though the scenarios it was
testing should probably be converted to integration tests. At a high level, the role of
`DirstatePersistence` has not changed, but the exact data it writes is much different.
Its corresponding unit test is also disabled, for now.
Note that this revision does not change the name of the file where "dirstate data" is written
(this is defined as `kDirstateFile` in `ClientConfig.cpp`), so we should blow away any existing
instances of this file once this change lands. (It is still early enough in the project that it does
not seem worth the overhead of a proper migration.)
The true test of the success of this new approach is the ease with which we can write more
integration tests for things like `hg histedit` and `hg graft`. Ideally, these should require very
few changes to `eden_dirstate.py`.
Reviewed By: simpkins
Differential Revision: D5071778
fbshipit-source-id: e8fec4d393035d80f36516ac050cad025dc3ba31
Summary:
This is generated by applying clang-tidy `-checks=modernize-use-override` to all the c++ code in project eden.
It enforces the use of the keywords `virtual`, `verride` and `final` in a way compliant to the style guide.
Reviewed By: igorsugak
Differential Revision: D5108807
fbshipit-source-id: 596f2d73f1137de350114416edb1c37da5423ed5
Summary: The `nodiscard` attribute was added to standard C++ in C++17. Prefer the standard formulation when it is available; otherwise, use `__attribute__((__warn_unused_result__))` on compilers that support the GNU extensions, and `_Check_return_` on MSVC. The old `FOLLY_WARN_UNUSED_RESULT` is now expands to `FOLLY_NODISCARD`.
Reviewed By: yfeldblum
Differential Revision: D5105137
fbshipit-source-id: 9aa22e81cd9f0b89f9343433aeae3ba365227ccb
Summary:
I was working on a new test and I got an error that `directaccess` must be
enabled for `inhibit` to work.
Reviewed By: simpkins
Differential Revision: D5077133
fbshipit-source-id: cc5235c845e3f299f96e1c901ef4aea18ca57b76
Summary:
We're using `window` to constrain resources and maintain an upper
bound on the number of futures that we spawn. Ironically, for a largish number
of already completed futures, this results in a recursive chain of `spawn`
calls in the `window` implementation and results in a stack overflow (the kind
that crashes a program, not the kind that answers programming questions).
This diff removes the use of `window` and replaces it with `collect`.
An earlier iteration of this diff just made all of these calls serially,
blocking the main thread. This at least does things in parallel; both
can torture the system for pathological walks, but at least the parallel
one should complete faster.
Reviewed By: simpkins
Differential Revision: D4932774
fbshipit-source-id: 40d39a85029f38ff69a530070efb879a81950451
Summary:
I had to add simple implementations to various things in `edendirstate`
in order to be able to run `hg histedit`. There is still a lot more to do, but
at least this gives us a starting point to iterate and a test to demonstrate
the most simple functionality.
Reviewed By: wez
Differential Revision: D5049308
fbshipit-source-id: 34727f633c003cacae44108eb3ece06590098c7b
Summary:
Note that we must specify quite a few extensions to get behavior that is
representative of how Hg works at Facebook.
Reviewed By: DurhamG
Differential Revision: D5057478
fbshipit-source-id: ee774a9b8dcebe82e4b19cc52f9b0b5a53e6420c
Summary:
Recall that we override `$HOME` in integration tests, so this will not overwrite
your personal `~/.hgrc` when you run an integration test.
An upcoming integration test for `hg histedit` that I am working on requires
this value to be set.
Reviewed By: wez
Differential Revision: D5051112
fbshipit-source-id: 2fd8541aa6504640b08337fdc22160e243beaae3
Summary:
`HgExtensionTestBase.assert_status()` was added in D4814422, but it was only
applied to `update_test.py`. This change updates the docstring (it appears to
have been copy/pasted from a nearby method), and makes use of it in the other
integration tests.
Reviewed By: wez
Differential Revision: D5050775
fbshipit-source-id: bb70740b6f455a84e7a22c3286c8ddbe2462f816
Summary:
Previous to this change, when I would add `import pudb; pudb.set_trace()` to do
some debugging, two annoying things would happen:
- I would have to edit the `TARGETS` file to add `pudb` as a dependency and
then `buck build eden/integration/hg` again.
- When I hit a breakpoint using `pudb`, I would have to go through the welcome
screen, change the theme, etc., because my settings were not found.
Now that I figured out what the problem was, I added instructions to the
`TARGETS` file to help others fall into the pit of success.
Reviewed By: wez
Differential Revision: D5050725
fbshipit-source-id: 1896f9f52eb056b3295b2d8e896dabb5d990ba22
Summary:
It is an error to not provide mode argument when open is
called with O_CREAT.
Reviewed By: oebeling
Differential Revision: D5020517
fbshipit-source-id: 87665ceb16ebdff797f8bf3d33bea082191e3121
Summary:
On OSX mercurial libraries are not installed in the standard pythonpath. This
option lets me pass a pythonpath to the hg importer script only, without
polluting the pythonpath for other subprocess invocations (such as python 3
based hooks, which are not happy with python2 pythonpaths).
Reviewed By: bolinfest
Differential Revision: D4992222
fbshipit-source-id: aa8e734970344215cc8783e164fd119c3ee6aed6
Summary:
These files are defining flags, so they should import gflags.
This import being missing causes open source build to fail.
Reviewed By: bolinfest
Differential Revision: D4992246
fbshipit-source-id: 61eb35b43ae6f9ff88bd34a9ebc8db6d80ac9d08
Summary:
When we say we are building "all" of Eden, we generally want to ensure we have
built the pyremote service that the Hg extension needs, as well.
Reviewed By: wez
Differential Revision: D4982118
fbshipit-source-id: 7282266e82e662a48b5ced6da6b7daedab8b65ea
Summary:
Update Dirstate::remove() to directly call TreeInode::unlink() instead of going
through the EdenDispatcher. Internal code should be able to use TreeInode
directly without going through the FUSE Dispatcher. This prevents us from
having to look up the TreeInode object by inode number again.
This also fixes TreeInode::unlink() and TreeInode::rmdir() to perform FUSE
cache invalidation correctly. EdenDispatcher used to do this for unlink()
calls, but it was missing for rmdir() calls.
Reviewed By: wez
Differential Revision: D4968832
fbshipit-source-id: 51fda5de196392c640436ca6ad7d8333e6799db9
Summary:
This replaces the shouldFileBeDeletedByHgRemove() function with a much simpler
and safer check.
It turns out this code was more complicated than it needed to be: it looked up
the inode in question and confirmed that it still existed, even though it's
calling function had already previously looked up the inode object.
I believe this deletes the last place in the code that still assumes that an
InodeBase object must be loaded if the inode has previously been materialized.
This also removes one of the last remaining places that directly holds the
TreeInode contents_ lock outside of TreeInode.cpp.
Reviewed By: bolinfest
Differential Revision: D4968834
fbshipit-source-id: a37bc0d7889eb81cca5b045cbc82732a53ef53d4
Summary:
This updates the Dirstate::addAll() code to use EdenMount::diff() internally,
instead of getStatusForExistingDirectory().
This lets us delete getStatusForExistingDirectory() and several other helper
functions that it used. The logic in getStatusForExistingDirectory() was based
on the incorrect assumption that files can only be modified from their expected
source control state if they are materialized. This could result in incorrect
results in a variety of cases (particularly after renames, or "hg reset"
operations). The EdenMount::diff() code does not have this problem.
That said, this new addAll() code does still have some performance issues--it
currently does a full tree diff for each input path. We should ideally fix
this in the future to only diff the necessary subtree for each path. However,
in the short term this trade-off seems worth being able to delete all of this
older, buggy code. diff() should be cheap enough in most cases that this won't
be a major problem unless a large number of paths are given as input.
Reviewed By: bolinfest
Differential Revision: D4968835
fbshipit-source-id: 1834aa98a26dcaa0e1c06c7ac25c57944fa1b5f7
Summary:
Make sure we use pointers to const GitIgnoreStack objects when possible.
Also change the code to store the root GitIgnoreStack object for a diff()
operation in the DiffContext object, instead of having EdenMount::diff()
allocate it separately on the heap. This will make it easier to consolidate a
bit more logic in the DiffContext class in the future. In particular, I
suspect we will want a version of diff() that only works on one portion of the
tree. Putting more of the functionality in DiffContext will make it slightly
easier to share code between the full-mount diff and the subtree diff
functions. This also happens to save a memory allocation for now.
Reviewed By: wez
Differential Revision: D4968833
fbshipit-source-id: 1dc33b3d44cdf00e93b22d810c3a736d27c13638
Summary:
Update the eden thrift APIs to support multiple thrift parents, and update the
hg extension to use the new thrift APIs.
Reviewed By: bolinfest
Differential Revision: D4949139
fbshipit-source-id: 98a7bd17ccad7be6a429df34ecfaf3fe7ae0b39a
