Summary:
Fortunately, this passed on the first try: it did not require any bug fixes in
Eden!
(Note: this ignores all push blocking failures!)
Reviewed By: simpkins
Differential Revision: D5698953
fbshipit-source-id: c5ce39725f8d14b5ea93bd3cafeb5e566f92d326
Summary:
Fortunately, this passed on the first try: it did not require any bug fixes in
Eden! Though admittedly, most of the relevant fixes were presumably done in
D5686114.
(Note: this ignores all push blocking failures!)
Reviewed By: simpkins
Differential Revision: D5696055
fbshipit-source-id: 0099db501ae1a5d72528d222dee0176fc1fc4332
Summary:
Update the integration test framework so that we can run the hg integration
tests with several different hg config settings, using different sets of
mercurial extensions.
This adds code to test using flat manifest, treemanifest in hybrid mode, and
treemanifest in tree only mode. However, the treeonly configuration is
disabled at the moment due to some bugs in treeonly behavior preventing it from
being able to create test repositories in treeonly mode.
Reviewed By: bolinfest
Differential Revision: D5685880
fbshipit-source-id: 081ead4e77cd14a7feb03381783395bd5a8fef4f
Summary:
The most recent mercurial release updated the 'successors()' revset to be the
same as 'allsuccessors()', and it always includes the argument itself in the
output now. This updates the revset to exclude the input commit.
Reviewed By: bolinfest
Differential Revision: D5694826
fbshipit-source-id: 3e931a39675262f33a5298701b4559e0d9906490
Summary:
This makes some minor tweaks to the behavior of the HgRepository.log() helper
function in the integration tests.
Previously this command did not take a revset argument, and instead relied on
the Facebook tweakdefaults extension to use the `--follow` behavior when no
revset was specified. (Without tweakdefaults mercurial uses `tip:0` by
default, which is not what the histedit tests expect.)
I added a revset parameter now, and updated it to default to `::.`. This is
close to the previous behavior, although I intentionally left it reporting
commits from oldest to newest now.
I also updated the log code to add its own delimiter to the template, rather
than requiring callers to always append an escaped nul byte to the template.
Reviewed By: bolinfest
Differential Revision: D5685876
fbshipit-source-id: 01578f62d553be1cd8002b5718d7f12a2f41d4d8
Summary:
Update the integration tests to no longer enable the evolve or fsmonitor
extensions in the test repositories.
evolve has been deprecated at Facebook for a while now and isn't even shipped
as part of our mercurial installation any more. This settings was just causing
a warning to be printed that this extension could not be found.
The fsmonitor extension also didn't have any real effect, even in the backing
repository. We don't create .watchmanconfig files in the test repositories, so
watchman won't watch them. Therefore fsmonitor simply printed warnings that
watchman wasn't watching this repository.
Reviewed By: bolinfest
Differential Revision: D5685879
fbshipit-source-id: 85b8a725bd17890a93be5c71dd5a0f3f1d744598
Summary:
Fix the integration tests to store hg config settings in the .hg/hgrc file in
the backing repository. Previously the tests saved settings to a temporary
file, and then always invoked hg with HGRCPATH pointing at this temporary file.
Unfortunately this resulted in the integration test code using different hg
settings than edenfs, since edenfs was never aware of this temporary file.
Defining the settings in the backing repository's normal .hg/hgrc file means
that edenfs will be able to see these settings as well. The eden post-clone
hooks will also automatically copy these settings in to the mount point, so
that we do not need to use a custom HGRCPATH setting inside the eden mount
either.
Reviewed By: bolinfest
Differential Revision: D5685877
fbshipit-source-id: 1857554d0cf1a585fe55577eb48a87686f9476ca
Summary:
Note that this feature was mostly implemented before this commit, but never
tested. Unsurprisingly, there were bugs.
This change also introduces a new `eden debug hg_copy_map_get_all` subcommand
because that was a straightforward way to verify the internal state of the copy
map on the server side from an integration test.
Adding this test uncovered a key copy/paste bug in `EdenThriftClient.py`
(`hgCopyMapGet` was being invoked instead of `hgCopyMapPut`.)
It also uncovered a bug in `LameThriftClient` because the `compile()` and
`eval()` calls on the output are not appropriate when the return type of the
Thrift endpoint is `string`.
Reviewed By: simpkins
Differential Revision: D5686114
fbshipit-source-id: f0093d2b67062c01982dc5bc1f0db2774b3a9356
Summary:
In `hg/eden/__init__.py`, we wrap `match()` in Mercurial's `match.py` in an
attempt to annotate every `basematcher` created in the system with a special
`_eden_match_info` property that we can use in `_eden_walk_helper()` to perform
walks more efficiently. Unfortunately, we missed a case where `scmutil.py`
has a `matchfiles()` function that calls `exact()` in `match.py` directly rather
than going through the generic `match()` function.
This was causing a failure when running `hg revert <filename>` in Eden because
the matcher that was created via `exact()` did not have an `_eden_match_info`.
This commit wraps `exact()` to add the property.
Reviewed By: wez
Differential Revision: D5674660
fbshipit-source-id: 16d1e7648ebd7a23b43b9b1200d3e284e5bc07b0
Summary:
We verify this behavior by adding the appropriate debug statements to our Hg
extension, running `hg update` with the `--debug` flag, and verifying the
expected log statements appear in stdout.
Reviewed By: simpkins
Differential Revision: D5505478
fbshipit-source-id: 389b69c7c1de4fa88fd07fb9eb4dd7e3370e766b
Summary:
This is a relatively simple test that takes two branches with no conflicts and
rebases one on top of the other. It also provides modest checks to ensure Eden
does not load a bunch of inodes unnecessarily when updating to the new head.
This also introduces `EdenServerInspector`, which provides convenience methods
for inspecting the Eden server via Thrift.
Reviewed By: simpkins
Differential Revision: D5504741
fbshipit-source-id: 6636c431658f24a850d0e5404d1a0e4f0528a781
Summary:
Up until now, we have largely ignored Mercurial's logic around transactions.
Previously, the `savebackup()` and `restorebackup()` methods of `eden_dirstate`
were no-ops, but with this change, we provide real implementations for
`savebackup()` and `restorebackup()`.
We no longer try to maintain the `_dirty` field in `eden_dirstate` as of this change
as we should not be exercising any superclass code that relies on its value.
Reviewed By: simpkins
Differential Revision: D5485950
fbshipit-source-id: effa5f83ead8c6eefa613c196d7ec5646da31e4b
Summary:
Added an integration test to verify behavior for `hg rm`. The test passed
without any changes to Eden required!
(Note: this ignores all push blocking failures!)
Reviewed By: simpkins
Differential Revision: D5494421
fbshipit-source-id: ca9578268adb353380e96aced6e2f1caca52b6a3
Summary:
Now `eden_dirstate` can inherit the implementations of the `branch()` and
`setbranch()` methods.
More importantly, we have decided to let `dirstate` do its natural management
of the `.hg` directory (with some exceptions, such as the contents of the
`.hg/dirstate` file) because there are interactions with `localrepo` and other
parts of Mercurial that use the filesystem as the API rather than an explicit
programmatic API, and we want to make sure we maintain the expected invariants.
(Note: this ignores all push blocking failures!)
Reviewed By: simpkins
Differential Revision: D5464059
fbshipit-source-id: 539e142ba11caea3e8487a3cc70d53cc236d2545
Summary:
This updates the TARGETS files so that we don't run all integration tests in
both the FB and OSS build flavors any more. For the OSS build we only run the
tests in basic_tests.py now.
This reduces the overall time to run all eden tests from about 170 seconds to
about 120 seconds on my server.
Reviewed By: wez
Differential Revision: D5367406
fbshipit-source-id: 81048f415e2f69175059f346e3f81269142a5fc7
Summary:
Update integration tests to pass logging settings in via the new `--logging`
argument instead of using `--vmodule`. Now that eden log messages have all
been switched from glog to folly logging, the `--logging` flag needs to be
used.
Reviewed By: wez
Differential Revision: D5367618
fbshipit-source-id: 299f73a8376863e46b7fffe59c539e54c78caed4
Summary:
Upstream mercurial recently added calls to dirstate.copies().copy(),
so update the dummy_copymap() class to support this.
Reviewed By: wez
Differential Revision: D5316773
fbshipit-source-id: a2454daf265faaf2bcf1a539c8bf428f8a5e9830
Summary:
1. Moved read, write, mkdir, rm methods in hg/lib/hg_extension_test_base.py to lib/test_case.py.
2. Added integration test case to test unload free inodes.
Reviewed By: simpkins
Differential Revision: D5277870
fbshipit-source-id: b93b6049a10357cf8c92366e6dca3968f7f30c30
Summary:
This is a major change to Eden's Hg extension.
Our initial attempt to implement `edendirstate` was to create a "clean room"
implementation that did not share code with `mercurial/dirstate.py`. This was
helpful in uncovering the subset of the dirstate API that matters for Eden. It
also provided a better safeguard against upstream changes to `dirstate.py` in
Mercurial itself.
In this implementation, the state transition management was mostly done
on the server in `Dirstate.cpp`. We also made a modest attempt to make
`Dirstate.cpp` "SCM-agnostic" such that the same APIs could be used for
Git at some point.
However, as we have tried to support more of the sophisticated functionality
in Mercurial, particularly `hg histedit`, achieving parity between the clean room
implementation and Mercurial's internals has become more challenging.
Ultimately, the clean room implementation is likely the right way to go for Eden,
but for now, we need to prioritize having feature parity with vanilla Hg when
using Eden. Once we have a more complete set of integration tests in place,
we can reimplement Eden's dirstate more aggressively to optimize things.
Fortunately, the [[ https://bitbucket.org/facebook/hg-experimental/src/default/sqldirstate/ | sqldirstate ]]
extension has already demonstrated that it is possible to provide a faithful
dirstate implementation that subclasses the original `dirstate` while using a different
storage mechanism. As such, I used `sqldirstate` as a model when implementing
the new `eden_dirstate` (distinguishing it from our v1 implementation, `edendirstate`).
In particular, `sqldirstate` uses SQL tables as storage for the following private fields
of `dirstate`: `_map`, `_dirs`, `_copymap`, `_filefoldmap`, `_dirfoldmap`. Because
`_filefoldmap` and `_dirfoldmap` exist to deal with case-insensitivity issues, we
do not support them in `eden_dirstate` and add code to ensure the codepaths that
would access them in `dirstate` never get exercised. Similarly, we also implemented
`eden_dirstate` so that it never accesses `_dirs`. (`_dirs` is a multiset of all directories in the
dirstate, which is an O(repo) data structure, so we do not want to maintain it in Eden.
It appears to be primarily used for checking whether a path to a file already exists in
the dirstate as a directory. We can protect against that in more efficient ways.)
That leaves only `_map` and `_copymap` to worry about. `_copymap` contains the set
of files that have been marked "copied" in the current dirstate, so it is fairly small and
can be stored on disk or in memory with little concern. `_map` is a bit trickier because
it is expected to have an entry for every file in the dirstate. In `sqldirstate`, it is stored
across two tables: `files` and `nonnormalfiles`. For Eden, we already represent the data
analogous to the `files` table in RocksDB/the overlay, so we do not need to create a new
equivalent to the `files` table. We do, however, need an equivalent to the `nonnormalfiles`
table, which we store in as Thrift-serialized data in an ordinary file along with the `_copymap`
data.
In our Hg extension, our implementation of `_map` is `eden_dirstate_map`, which is defined
in a Python file of the same name. Our implementation of `_copymap` is `dummy_copymap`,
which is defined in `eden_dirstate.py`. Both of these collections are simple pass-through data
structures that translate their method calls to Thrift server calls. I expect we will want to
optimize this in the future via some client-side caching, as well as creating batch APIs for talking
to the server via Thrift.
One advantage of this new implementation is that it enables us to delete
`eden/hg/eden/overrides.py`, which overrode the entry points for `hg add` and `hg remove`.
Between the recent implementation of `dirstate.walk()` for Eden and this switch
to the real dirstate, we can now use the default implementation of `hg add` and `hg remove`
(although we have to play some tricks, like in the implementation of `eden_dirstate.status()`
in order to make `hg remove` work).
In the course of doing this revision, I discovered that I had to make a minor fix to
`EdenMatchInfo.make_glob_list()` because `hg add foo` was being treated as
`hg add foo/**/*` even when `foo` was just a file (as opposed to a directory), in which
case the glob was not matching `foo`!
I also had to do some work in `eden_dirstate.status()` in which the `match` argument
was previously largely ignored. It turns out that `dirstate.py` uses `status()` for a number
of things with the `match` specified as a filter, so the output of `status()` must be filtered
by `match` accordingly. Ultimately, this seems like work that would be better done on the
server, but for simplicity, we're just going to do it in Python, for now.
For the reasons explained above, this revision deletes a lot of code `Dirstate.cpp`.
As such, `DirstateTest.cpp` does not seem worth refactoring, though the scenarios it was
testing should probably be converted to integration tests. At a high level, the role of
`DirstatePersistence` has not changed, but the exact data it writes is much different.
Its corresponding unit test is also disabled, for now.
Note that this revision does not change the name of the file where "dirstate data" is written
(this is defined as `kDirstateFile` in `ClientConfig.cpp`), so we should blow away any existing
instances of this file once this change lands. (It is still early enough in the project that it does
not seem worth the overhead of a proper migration.)
The true test of the success of this new approach is the ease with which we can write more
integration tests for things like `hg histedit` and `hg graft`. Ideally, these should require very
few changes to `eden_dirstate.py`.
Reviewed By: simpkins
Differential Revision: D5071778
fbshipit-source-id: e8fec4d393035d80f36516ac050cad025dc3ba31
Summary:
I was working on a new test and I got an error that `directaccess` must be
enabled for `inhibit` to work.
Reviewed By: simpkins
Differential Revision: D5077133
fbshipit-source-id: cc5235c845e3f299f96e1c901ef4aea18ca57b76
Summary:
I had to add simple implementations to various things in `edendirstate`
in order to be able to run `hg histedit`. There is still a lot more to do, but
at least this gives us a starting point to iterate and a test to demonstrate
the most simple functionality.
Reviewed By: wez
Differential Revision: D5049308
fbshipit-source-id: 34727f633c003cacae44108eb3ece06590098c7b
Summary:
Note that we must specify quite a few extensions to get behavior that is
representative of how Hg works at Facebook.
Reviewed By: DurhamG
Differential Revision: D5057478
fbshipit-source-id: ee774a9b8dcebe82e4b19cc52f9b0b5a53e6420c
Summary:
Recall that we override `$HOME` in integration tests, so this will not overwrite
your personal `~/.hgrc` when you run an integration test.
An upcoming integration test for `hg histedit` that I am working on requires
this value to be set.
Reviewed By: wez
Differential Revision: D5051112
fbshipit-source-id: 2fd8541aa6504640b08337fdc22160e243beaae3
Summary:
`HgExtensionTestBase.assert_status()` was added in D4814422, but it was only
applied to `update_test.py`. This change updates the docstring (it appears to
have been copy/pasted from a nearby method), and makes use of it in the other
integration tests.
Reviewed By: wez
Differential Revision: D5050775
fbshipit-source-id: bb70740b6f455a84e7a22c3286c8ddbe2462f816
Summary:
Previous to this change, when I would add `import pudb; pudb.set_trace()` to do
some debugging, two annoying things would happen:
- I would have to edit the `TARGETS` file to add `pudb` as a dependency and
then `buck build eden/integration/hg` again.
- When I hit a breakpoint using `pudb`, I would have to go through the welcome
screen, change the theme, etc., because my settings were not found.
Now that I figured out what the problem was, I added instructions to the
`TARGETS` file to help others fall into the pit of success.
Reviewed By: wez
Differential Revision: D5050725
fbshipit-source-id: 1896f9f52eb056b3295b2d8e896dabb5d990ba22
Summary:
This fixes "hg commit" so that it correctly updates the in-memory snapshot.
This has been broken ever since I added the in-memory snapshot when
implementing checkout(). The existing scmMarkCommitted() method updated only
the Dirstate object and the on-disk SNAPSHOT file.
This diff fixes checkout() and resetCommit() to clear the Dirstate user
directives correctly, and then replaces calls to scmMarkCommitted() with
resetCommit().
Reviewed By: bolinfest
Differential Revision: D4935943
fbshipit-source-id: 5ffcfd5db99f30c730ede202c5e013afa682bac9
Summary:
This updates how we build and package the eden hg extension, and how we find it
during integration tests.
- Update the extension to always look relative to its current location to find
the other modules it depends on. This ensures that the integration tests
always find modules from the local repository, and do not use the modules
installed on the system.
- Add a buck rule to unpack the python archive at build time. This is needed
for integration tests to use the local version of the module.
- Ensure that we install a correct `hgext3rd/__init__.py` module in the eden
extension directory. This is required to correctly set up `hgext3rd` as a
namespace package. This also unfortunately needs to be a `.py` file, and not
just a .pyc file. (The pkgutil.expand_path() code looks specifically for
directories containing `__init__.py` files, and does not check for
`__init__.pyc`.)
- Update the extension to only try importing the native thrift modules if we
are running python 2.7.6 or greater. Python 2.7.6 is the first that supports
unicode arguments to `struct.pack()`, which thrift requires. Python 2.7.5 can
import the thrift modules, but throws errors when trying to run them.
Reviewed By: bolinfest
Differential Revision: D4935279
fbshipit-source-id: 9af81736124c55476a5eb5beba9474a4371a639b
Summary:
Fix a subtle crash during checkout when handling newly added entries that
already exist in the working directory: CheckoutAction passed the entry name to
checkoutUpdateEntry() as a PathComponentPiece. However, this
PathComponentPiece could refer to the entry name owned by newScmEntry_, and it
also passed newScmEntry_ into checkoutUpdateEntry() as an rvalue reference.
As a result, if the string data was stored invalidated by the move the name
would no longer be valid when checkoutUpdateEntry() tried to use it.
This bug is triggered by doing an "hg update --clean", where a file added in
the destination commit already exists on disk, and has an entry name of 23
characters or less. (The 23 character limit is fbstring's upper bound on
small string optimizations, where it will store the string data inline in the
object, causing it to be invalidated on move.)
This also fixes a crash in a VLOG() statement when the verbose log level for
TreeInode.cpp was set to 4 or greater.
Reviewed By: bolinfest
Differential Revision: D4882544
fbshipit-source-id: 917ede6eeae2224aaa0724b8b30324f3c3a5c924
Summary:
Update the hg extension to implement dirstate.rebuild(). This is necessary for
the `hg reset` command. This also now implements dirstate.setparents() for
cases when there is only one parent.
Reviewed By: wez
Differential Revision: D4823780
fbshipit-source-id: 802de006e03860995095dc3af17acb2eb05f4e8b
Summary:
Add an assert_status() method to the hg integration tests that runs "hg
status", parses the output, then compares it to expected results.
Reviewed By: wez
Differential Revision: D4814422
fbshipit-source-id: 24ebdc2e0239c4833953c31e5786cc320bcd9d62
Summary:
The hg_import_helper script that eden uses to import data from mercurial keeps
a long-lived repository object open. This caches some data about the
repository, and if new commits are added after it was created, it can fail to
see them.
This updates hg_import_helper.py to catch errors that occur when trying to use
the repository objects. The code will invalidate the repository object and
then retry the operation once, in the hopes that it will now succeed after
invalidation.
Reviewed By: bolinfest
Differential Revision: D4752659
fbshipit-source-id: 1c75c84766d6bbda0710882a338eaa09e0cb0030
Summary:
The kernel can return ENOENT in response to invalidation notification if we
have never told the kernel about the inode in question. This resulted in
spurious errors during checkout when updating files that were loaded internally
by edenfs rather than via FUSE call. For instance, this was commonly triggered
by .gitignore files, which eden loads on its own to perform ignore processing.
Reviewed By: bolinfest
Differential Revision: D4752630
fbshipit-source-id: d4e092643a8d33cf33709f7e3664289f167ac093
Summary:
I found it rather awkward in HgExtensionTestBase that self.repo is not actually
the repository being tested. It was instead the repostiory used as the backing
store for the mercurial data, and self.repo_for_mount was the repository being
tested.
This diff renames the two repository classes, so that self.backing_repo is now
the backing store repository, and self.repo is the repository being tested.
In order to do this I changed HgExtensionTestBase to derive directly from
EdenTestCase. Previously it derived from EdenHgTest, and was letting
EdenHgTest set up self.repo. It seemed more understandable to avoid deriving
EdenHgTest now since self.repo is not the repository that needs to be set up
initially.
Reviewed By: bolinfest
Differential Revision: D4752631
fbshipit-source-id: d8b542b0ecead66b965af1a582085345e28b2908
Summary:
Previously the eden hg extension short-circuited the checkout operation if the
destination commit was the same as the one currently checked out. This was
incorrect if --clean was specified, since we do need to reset the working
directory state in this case.
This updates the extension code to always make the thrift checkout() call when
doing a force checkout.
This also avoids calling applyupdates() to resolve conflicts when force=True.
When doing a force checkout, eden reports files with conflicts that it
overwrote, but these do not need to be resolved by mercurial.
Additionally, this also updates a couple other APIs that have recently been
changed in upstream mercurial: merge.update() now takes an updatecheck
argument, and repo.join() should now be written as repo.vfs.join().
Reviewed By: bolinfest
Differential Revision: D4752510
fbshipit-source-id: e1ee92d086315e35a1378f674e668876a667c0ce
Summary:
Move the integration tests from eden/fs/integration up one directory, to
eden/integration.
The main benefit is that this makes it easy to run just the edenfs unit tests
by running "buck test eden/fs/...". These unit tests complete much more
quickly than the full set of integration tests, providing a faster test suite
to re-run repeatedly during development. The integration tests can be run with
"buck test eden/integration/...", and the full set of tests can still be run
with "buck test eden/..."
Reviewed By: wez
Differential Revision: D4490247
fbshipit-source-id: 5ceb5a19526f56e1cb926f352fa30ad2f1212c05
