Commit Graph

651 Commits

Author SHA1 Message Date
Adam Simpkins
8327c003e4 update integration tests to edit the repo .hgrc file
Summary:
Fix the integration tests to store hg config settings in the .hg/hgrc file in
the backing repository.  Previously the tests saved settings to a temporary
file, and then always invoked hg with HGRCPATH pointing at this temporary file.

Unfortunately this resulted in the integration test code using different hg
settings than edenfs, since edenfs was never aware of this temporary file.

Defining the settings in the backing repository's normal .hg/hgrc file means
that edenfs will be able to see these settings as well.  The eden post-clone
hooks will also automatically copy these settings in to the mount point, so
that we do not need to use a custom HGRCPATH setting inside the eden mount
either.

Reviewed By: bolinfest

Differential Revision: D5685877

fbshipit-source-id: 1857554d0cf1a585fe55577eb48a87686f9476ca
2017-08-23 15:06:47 -07:00
Michael Bolin
0508c0b819 Change eden debug getpath to use pwd as the default mount point.
Summary: This seems a little more user-friendly.

Reviewed By: bradenwatling

Differential Revision: D5686562

fbshipit-source-id: 8142fb9105a3a44823f935fc04187cf0ed2258d7
2017-08-23 11:27:22 -07:00
Michael Bolin
303655c4b1 Add integration test for hg copy.
Summary:
Note that this feature was mostly implemented before this commit, but never
tested. Unsurprisingly, there were bugs.

This change also introduces a new `eden debug hg_copy_map_get_all` subcommand
because that was a straightforward way to verify the internal state of the copy
map on the server side from an integration test.

Adding this test uncovered a key copy/paste bug in `EdenThriftClient.py`
(`hgCopyMapGet` was being invoked instead of `hgCopyMapPut`.)

It also uncovered a bug in `LameThriftClient` because the `compile()` and
`eval()` calls on the output are not appropriate when the return type of the
Thrift endpoint is `string`.

Reviewed By: simpkins

Differential Revision: D5686114

fbshipit-source-id: f0093d2b67062c01982dc5bc1f0db2774b3a9356
2017-08-22 21:06:07 -07:00
Jyothsna Konisa
72b61a5ddc Changes to return unloaded inode count for TreeInode::unloadChildrenNow
Summary:
1.Modified `TreeInode::unloadChildrenNow()` to return number of inodes that have been unloaded.
2.Modified `EdenServiceHandler::unloadInodeForPath()` to return number of inodes that are unloaded.

Reviewed By: simpkins

Differential Revision: D5627539

fbshipit-source-id: 4cdb0433dced6bf101158b9e6f8c35de67d9abbe
2017-08-22 19:50:00 -07:00
Jyothsna Konisa
371cfa097d TestCase to verify unloadChildrenNow with age
Summary:
Added a test case `test_unload_free_inodes_age` to verify the behaviour of unloadChildrenNow with age parameter.
Added new parameter age to `unloadInodeForPath` in eden.thrift, and `EdenServiceHandler`.
Modified `do_unload_inodes` function in `debug.py` to support the new behaviour.

Reviewed By: simpkins

Differential Revision: D5565859

fbshipit-source-id: a35053725be26bc906cf158969cbe21db1cbadde
2017-08-22 19:50:00 -07:00
Michael Bolin
f946ae76f9 Wrap exact() from Mercurial's match.py.
Summary:
In `hg/eden/__init__.py`, we wrap `match()` in Mercurial's `match.py` in an
attempt to annotate every `basematcher` created in the system with a special
`_eden_match_info` property that we can use in `_eden_walk_helper()` to perform
walks more efficiently. Unfortunately, we missed a case where `scmutil.py`
has a `matchfiles()` function that calls `exact()` in `match.py` directly rather
than going through the generic `match()` function.

This was causing a failure when running `hg revert <filename>` in Eden because
the matcher that was created via `exact()` did not have an `_eden_match_info`.
This commit wraps `exact()` to add the property.

Reviewed By: wez

Differential Revision: D5674660

fbshipit-source-id: 16d1e7648ebd7a23b43b9b1200d3e284e5bc07b0
2017-08-21 18:05:03 -07:00
Simon Jensen
3a0e66677b Cache invalidation thrift api
Summary: Provide a thrift interface to invalidate the cache for an inode denoted by path.

Reviewed By: simpkins

Differential Revision: D5655387

fbshipit-source-id: 887aa4963d216a0d8eed93b6fb8721632cc31d19
2017-08-21 16:05:44 -07:00
Michael Bolin
70050affcc Introduce eden debug hg_get_dirstate_tuple.
Summary:
This is a convenient way to test the `hgGetDirstateTuple()` endpoint in
`eden.thrift`.

Reviewed By: quark-zju, wez

Differential Revision: D5654237

fbshipit-source-id: 0b285e056002d4556733a53293582345f36780b2
2017-08-18 21:49:59 -07:00
Braden Watling
cf297e0106 Add subcommand getpath to eden debug
Summary: Add a command to lookup the path for an inode given the inode number and eden mount path.

Reviewed By: bolinfest

Differential Revision: D5627411

fbshipit-source-id: 25928f506d3f48d8a6784fe81fb17fa0500d6bc9
2017-08-16 20:04:30 -07:00
Jyothsna Konisa
916c129655 setting TimeStamps for TreeInode
Summary:
Updated time stamps of TreeInode accurately on mkdir,rmdir,mknode,symlink,create,unlink and readdir.
updated the `TreeInode::getattr` function to return in-memory timestamps.

Reviewed By: simpkins

Differential Revision: D5568183

fbshipit-source-id: c36f7fb767cd4342aab5cc983eea56e37cd2077e
2017-08-14 23:23:23 -07:00
Jyothsna Konisa
8bcd0f234c Setting TimeStamps for FileInode
Summary:
updating atime,ctime,mtime of FileInode on read, write and setattr system calls.
modified `FileInode::stat` function to return accurate inmemory timestamps.

Reviewed By: simpkins

Differential Revision: D5552666

fbshipit-source-id: 86d446f72908663f8db509b7b789d9f35d17df3a
2017-08-14 23:23:23 -07:00
Wez Furlong
c08890f849 do a better job at reporting "new" in watchman results.
Summary:
We're seeing that this is always set to true for eden,
which is causing buck to run slower than it should.

To make this work correctly, I've augmented our journal data structure
so that it can track create, change and remove events for the various
paths.

I've also plumbed rename events into the journal.

This requires a slightly more complex merge routine, so I've refactored the two
call sites that were merging in slightly different contexts so that they can
now share the same guts of the merge routine.  Perhaps slightly
counterintuitive in the merge code is that we merge a record from the past into
the state for now and this is a bit backwards compared to how people think.

I've expanded the eden integration test to check that we don't mix up
create/change/removes for the same path in a given window.

On the watchman side, we use the presence of the filename in the createdPaths
set as a hint that the file is new.  In that case we will set the watchman
`ctime` (which is not the state ctime but is really the *created clock time*)
to match the current journal position if the file is new, or leave it set
to 0 if the file is not known to be new.  This will cause the `is_new`
flag to be set appropriately by the code in `watchman/query/eval.cpp`;
if the sequence is 0 then it should never be set to true.  Otherwise (when
the file was in the `createPaths` set) it will be set to the current journal
position and this will be seen as newer than the `since` constraint on
the query and cause the file to show as `new`.

Reviewed By: bolinfest

Differential Revision: D5608538

fbshipit-source-id: 8d78f7da05e5e53110108aca220c3a97794f8cc2
2017-08-11 12:57:37 -07:00
Michael Bolin
c8faca5285 Updated RebaseTest to verify that Eden's fast path was exercised.
Summary:
We verify this behavior by adding the appropriate debug statements to our Hg
extension, running `hg update` with the `--debug` flag, and verifying the
expected log statements appear in stdout.

Reviewed By: simpkins

Differential Revision: D5505478

fbshipit-source-id: 389b69c7c1de4fa88fd07fb9eb4dd7e3370e766b
2017-07-27 17:24:01 -07:00
Michael Bolin
31eac649e2 Add integration test for rebasing in Hg.
Summary:
This is a relatively simple test that takes two branches with no conflicts and
rebases one on top of the other. It also provides modest checks to ensure Eden
does not load a bunch of inodes unnecessarily when updating to the new head.

This also introduces `EdenServerInspector`, which provides convenience methods
for inspecting the Eden server via Thrift.

Reviewed By: simpkins

Differential Revision: D5504741

fbshipit-source-id: 6636c431658f24a850d0e5404d1a0e4f0528a781
2017-07-27 17:24:01 -07:00
Michael Bolin
b9b9ba32e9 Implement savebackup() and restorebackup() in eden_dirstate.
Summary:
Up until now, we have largely ignored Mercurial's logic around transactions.
Previously, the `savebackup()` and `restorebackup()` methods of `eden_dirstate`
were no-ops, but with this change, we provide real implementations for
`savebackup()` and `restorebackup()`.

We no longer try to maintain the `_dirty` field in `eden_dirstate` as of this change
as we should not be exercising any superclass code that relies on its value.

Reviewed By: simpkins

Differential Revision: D5485950

fbshipit-source-id: effa5f83ead8c6eefa613c196d7ec5646da31e4b
2017-07-27 17:24:01 -07:00
Michael Bolin
f91e7b07ec Created integration test for hg rm.
Summary:
Added an integration test to verify behavior for `hg rm`. The test passed
without any changes to Eden required!

(Note: this ignores all push blocking failures!)

Reviewed By: simpkins

Differential Revision: D5494421

fbshipit-source-id: ca9578268adb353380e96aced6e2f1caca52b6a3
2017-07-26 22:08:09 -07:00
Michael Bolin
401c69760d Pass a proper opener object to the parent constructor in eden_dirstate.
Summary:
Now `eden_dirstate` can inherit the implementations of the `branch()` and
`setbranch()` methods.

More importantly, we have decided to let `dirstate` do its natural management
of the `.hg` directory (with some exceptions, such as the contents of the
`.hg/dirstate` file) because there are interactions with `localrepo` and other
parts of Mercurial that use the filesystem as the API rather than an explicit
programmatic API, and we want to make sure we maintain the expected invariants.

(Note: this ignores all push blocking failures!)

Reviewed By: simpkins

Differential Revision: D5464059

fbshipit-source-id: 539e142ba11caea3e8487a3cc70d53cc236d2545
2017-07-25 11:58:26 -07:00
Adam Simpkins
128787b01b allow the hg_import_helper script to access pending commits
Summary:
This updates hg_import_helper.py to override
`mercurial.txnutil._mayhavepending()` to always return True.  This allows it
access to commits that are still part of a pending transaction, as long as
`writepending()` has been called on the transaction.  (We could have set the
`HG_PENDING` environment variable to the repository root path instead of
monkey-patching `_mayhavepending()`, but this felt more fragile--it requires
normalizing the repository root the same way that mercurial does, and making
sure we get the correct repository root when using the share extension.)

This allows us to remove our override of `commitctx.markcommitted()` in the hg
extension.  Previously we replaced this method to defer calling
`dirstate.setparents()` until the transaction was complete.  We no longer have
to do this since eden can now access the pending commit.

This should also help fix bugs in the rebase workflow where we previously
failed trying to checkout a still-pending commit.

Reviewed By: wez

Differential Revision: D5345451

fbshipit-source-id: 9c32ab655254c79f030c10b7c9d02563decb04f7
2017-07-07 18:45:02 -07:00
Adam Simpkins
8504688aaf don't run all integration tests in both build modes
Summary:
This updates the TARGETS files so that we don't run all integration tests in
both the FB and OSS build flavors any more.  For the OSS build we only run the
tests in basic_tests.py now.

This reduces the overall time to run all eden tests from about 170 seconds to
about 120 seconds on my server.

Reviewed By: wez

Differential Revision: D5367406

fbshipit-source-id: 81048f415e2f69175059f346e3f81269142a5fc7
2017-07-07 16:01:33 -07:00
Adam Simpkins
d214153a34 add some extra logging in integration tests
Summary:
Add some extra logging to the eden integration to report event timing during
tests.  For now I have added some basic logging during set up and tear down.

Test setup is pretty slow for most of our tests.  It looks like some of this is
spent waiting for the eden daemon to start up, but a lot of it is just waiting
on hg commands.  A lot of tests currently make separate "hg add" calls for each
file they create during repository setup, and this is unfortunately really
slow.  Changing these tests to make fewer "hg add" calls would probably speed
things up a fair amount.

Reviewed By: wez

Differential Revision: D5367620

fbshipit-source-id: 99270c1dc4f072b7a0c0eaed7be7730e78b9bb8c
2017-07-07 15:43:42 -07:00
Adam Simpkins
b0dde83ff2 update logging flags in integration tests
Summary:
Update integration tests to pass logging settings in via the new `--logging`
argument instead of using `--vmodule`.  Now that eden log messages have all
been switched from glog to folly logging, the `--logging` flag needs to be
used.

Reviewed By: wez

Differential Revision: D5367618

fbshipit-source-id: 299f73a8376863e46b7fffe59c539e54c78caed4
2017-07-07 15:43:42 -07:00
Adam Simpkins
d5b624e678 update dummy_copymap to work with latest upstream mercurial
Summary:
Upstream mercurial recently added calls to dirstate.copies().copy(),
so update the dummy_copymap() class to support this.

Reviewed By: wez

Differential Revision: D5316773

fbshipit-source-id: a2454daf265faaf2bcf1a539c8bf428f8a5e9830
2017-06-26 13:20:26 -07:00
Jyothsna Konisa
299f7d1373 Integration Test for UnnloadFreeInodes
Summary:
1. Moved read, write, mkdir, rm methods in hg/lib/hg_extension_test_base.py to lib/test_case.py.
2. Added integration test case to test unload free inodes.

Reviewed By: simpkins

Differential Revision: D5277870

fbshipit-source-id: b93b6049a10357cf8c92366e6dca3968f7f30c30
2017-06-22 22:38:47 -07:00
Jyothsna Konisa
ef1509c49b integration tests for timestamps
Summary:
1. Added timestamp integration tests for directories.
2. Modified few existing tests.
3. Added tests for cheking timestamps for open files.

Reviewed By: simpkins

Differential Revision: D5294138

fbshipit-source-id: 013c086644b03d76e2df43523c4b7aef657484ca
2017-06-21 23:22:08 -07:00
Andrew Gallagher
03bdaff954 codemod: format TARGETS with buildifier [4/5] (D5092623)
Reviewed By: igorsugak

fbshipit-source-id: 277a9d2bdc1d7e3ff3075bfe2d7307502fd0a507
2017-06-01 17:52:40 -07:00
Michael Bolin
57f5d72a27 Reimplement dirstate used by Eden's Hg extension as a subclass of Hg's dirstate.
Summary:
This is a major change to Eden's Hg extension.

Our initial attempt to implement `edendirstate` was to create a "clean room"
implementation that did not share code with `mercurial/dirstate.py`. This was
helpful in uncovering the subset of the dirstate API that matters for Eden. It
also provided a better safeguard against upstream changes to `dirstate.py` in
Mercurial itself.

In this implementation, the state transition management was mostly done
on the server in `Dirstate.cpp`. We also made a modest attempt to make
`Dirstate.cpp` "SCM-agnostic" such that the same APIs could be used for
Git at some point.

However, as we have tried to support more of the sophisticated functionality
in Mercurial, particularly `hg histedit`, achieving parity between the clean room
implementation and Mercurial's internals has become more challenging.
Ultimately, the clean room implementation is likely the right way to go for Eden,
but for now, we need to prioritize having feature parity with vanilla Hg when
using Eden. Once we have a more complete set of integration tests in place,
we can reimplement Eden's dirstate more aggressively to optimize things.

Fortunately, the [[ https://bitbucket.org/facebook/hg-experimental/src/default/sqldirstate/ | sqldirstate ]]
extension has already demonstrated that it is possible to provide a faithful
dirstate implementation that subclasses the original `dirstate` while using a different
storage mechanism. As such, I used `sqldirstate` as a model when implementing
the new `eden_dirstate` (distinguishing it from our v1 implementation, `edendirstate`).

In particular, `sqldirstate` uses SQL tables as storage for the following private fields
of `dirstate`: `_map`, `_dirs`, `_copymap`, `_filefoldmap`, `_dirfoldmap`. Because
`_filefoldmap` and `_dirfoldmap` exist to deal with case-insensitivity issues, we
do not support them in `eden_dirstate` and add code to ensure the codepaths that
would access them in `dirstate` never get exercised. Similarly, we also implemented
`eden_dirstate` so that it never accesses `_dirs`. (`_dirs` is a multiset of all directories in the
dirstate, which is an O(repo) data structure, so we do not want to maintain it in Eden.
It appears to be primarily used for checking whether a path to a file already exists in
the dirstate as a directory. We can protect against that in more efficient ways.)

That leaves only `_map` and `_copymap` to worry about. `_copymap` contains the set
of files that have been marked "copied" in the current dirstate, so it is fairly small and
can be stored on disk or in memory with little concern. `_map` is a bit trickier because
it is expected to have an entry for every file in the dirstate. In `sqldirstate`, it is stored
across two tables: `files` and `nonnormalfiles`. For Eden, we already represent the data
analogous to the `files` table in RocksDB/the overlay, so we do not need to create a new
equivalent to the `files` table. We do, however, need an equivalent to the `nonnormalfiles`
table, which we store in as Thrift-serialized data in an ordinary file along with the `_copymap`
data.

In our Hg extension, our implementation of `_map` is `eden_dirstate_map`, which is defined
in a Python file of the same name. Our implementation of `_copymap` is `dummy_copymap`,
which is defined in `eden_dirstate.py`. Both of these collections are simple pass-through data
structures that translate their method calls to Thrift server calls. I expect we will want to
optimize this in the future via some client-side caching, as well as creating batch APIs for talking
to the server via Thrift.

One advantage of this new implementation is that it enables us to delete
`eden/hg/eden/overrides.py`, which overrode the entry points for `hg add` and `hg remove`.
Between the recent implementation of `dirstate.walk()` for Eden and this switch
to the real dirstate, we can now use the default implementation of `hg add` and `hg remove`
(although we have to play some tricks, like in the implementation of `eden_dirstate.status()`
in order to make `hg remove` work).

In the course of doing this revision, I discovered that I had to make a minor fix to
`EdenMatchInfo.make_glob_list()` because `hg add foo` was being treated as
`hg add foo/**/*` even when `foo` was just a file (as opposed to a directory), in which
case the glob was not matching `foo`!

I also had to do some work in `eden_dirstate.status()` in which the `match` argument
was previously largely ignored. It turns out that `dirstate.py` uses `status()` for a number
of things with the `match` specified as a filter, so the output of `status()` must be filtered
by `match` accordingly. Ultimately, this seems like work that would be better done on the
server, but for simplicity, we're just going to do it in Python, for now.

For the reasons explained above, this revision deletes a lot of code `Dirstate.cpp`.
As such, `DirstateTest.cpp` does not seem worth refactoring, though the scenarios it was
testing should probably be converted to integration tests. At a high level, the role of
`DirstatePersistence` has not changed, but the exact data it writes is much different.
Its corresponding unit test is also disabled, for now.

Note that this revision does not change the name of the file where "dirstate data" is written
(this is defined as `kDirstateFile` in `ClientConfig.cpp`), so we should blow away any existing
instances of this file once this change lands. (It is still early enough in the project that it does
not seem worth the overhead of a proper migration.)

The true test of the success of this new approach is the ease with which we can write more
integration tests for things like `hg histedit` and `hg graft`. Ideally, these should require very
few changes to `eden_dirstate.py`.

Reviewed By: simpkins

Differential Revision: D5071778

fbshipit-source-id: e8fec4d393035d80f36516ac050cad025dc3ba31
2017-05-26 12:05:29 -07:00
Michael Bolin
33ec2b7526 Add directaccess to the list of extensions we enable during integration tests.
Summary:
I was working on a new test and I got an error that `directaccess` must be
enabled for `inhibit` to work.

Reviewed By: simpkins

Differential Revision: D5077133

fbshipit-source-id: cc5235c845e3f299f96e1c901ef4aea18ca57b76
2017-05-18 20:35:36 -07:00
Michael Bolin
2286e9fb14 Perform a basic histedit command and add an integration test.
Summary:
I had to add simple implementations to various things in `edendirstate`
in order to be able to run `hg histedit`. There is still a lot more to do, but
at least this gives us a starting point to iterate and a test to demonstrate
the most simple functionality.

Reviewed By: wez

Differential Revision: D5049308

fbshipit-source-id: 34727f633c003cacae44108eb3ece06590098c7b
2017-05-16 12:36:17 -07:00
Michael Bolin
145ca986a7 Specify HGRCPATH for Hg integration tests to make test environment hermetic.
Summary:
Note that we must specify quite a few extensions to get behavior that is
representative of how Hg works at Facebook.

Reviewed By: DurhamG

Differential Revision: D5057478

fbshipit-source-id: ee774a9b8dcebe82e4b19cc52f9b0b5a53e6420c
2017-05-15 11:11:00 -07:00
Michael Bolin
85a107c550 Set ui.username in ~/.hgrc in integration tests.
Summary:
Recall that we override `$HOME` in integration tests, so this will not overwrite
your personal `~/.hgrc` when you run an integration test.

An upcoming integration test for `hg histedit` that I am working on requires
this value to be set.

Reviewed By: wez

Differential Revision: D5051112

fbshipit-source-id: 2fd8541aa6504640b08337fdc22160e243beaae3
2017-05-12 13:05:50 -07:00
Michael Bolin
f766fe0a87 Update Hg integration tests to use assert_status().
Summary:
`HgExtensionTestBase.assert_status()` was added in D4814422, but it was only
applied to `update_test.py`. This change updates the docstring (it appears to
have been copy/pasted from a nearby method), and makes use of it in the other
integration tests.

Reviewed By: wez

Differential Revision: D5050775

fbshipit-source-id: bb70740b6f455a84e7a22c3286c8ddbe2462f816
2017-05-12 09:19:57 -07:00
Michael Bolin
460ec77a8b Add pudb to the integration test dependencies for ease of use.
Summary:
Previous to this change, when I would add `import pudb; pudb.set_trace()` to do
some debugging, two annoying things would happen:

- I would have to edit the `TARGETS` file to add `pudb` as a dependency and
then `buck build eden/integration/hg` again.
- When I hit a breakpoint using `pudb`, I would have to go through the welcome
screen, change the theme, etc., because my settings were not found.

Now that I figured out what the problem was, I added instructions to the
`TARGETS` file to help others fall into the pit of success.

Reviewed By: wez

Differential Revision: D5050725

fbshipit-source-id: 1896f9f52eb056b3295b2d8e896dabb5d990ba22
2017-05-12 09:19:57 -07:00
Adam Simpkins
87cbfe142b update the in-memory snapshot correctly after a commit
Summary:
This fixes "hg commit" so that it correctly updates the in-memory snapshot.
This has been broken ever since I added the in-memory snapshot when
implementing checkout().  The existing scmMarkCommitted() method updated only
the Dirstate object and the on-disk SNAPSHOT file.

This diff fixes checkout() and resetCommit() to clear the Dirstate user
directives correctly, and then replaces calls to scmMarkCommitted() with
resetCommit().

Reviewed By: bolinfest

Differential Revision: D4935943

fbshipit-source-id: 5ffcfd5db99f30c730ede202c5e013afa682bac9
2017-04-24 18:06:59 -07:00
Adam Simpkins
5da361f55b improve building and importing of the eden hg extension
Summary:
This updates how we build and package the eden hg extension, and how we find it
during integration tests.

- Update the extension to always look relative to its current location to find
  the other modules it depends on.  This ensures that the integration tests
  always find modules from the local repository, and do not use the modules
  installed on the system.

- Add a buck rule to unpack the python archive at build time.  This is needed
  for integration tests to use the local version of the module.

- Ensure that we install a correct `hgext3rd/__init__.py` module in the eden
  extension directory.  This is required to correctly set up `hgext3rd` as a
  namespace package.  This also unfortunately needs to be a `.py` file, and not
  just a .pyc file.  (The pkgutil.expand_path() code looks specifically for
  directories containing `__init__.py` files, and does not check for
  `__init__.pyc`.)

- Update the extension to only try importing the native thrift modules if we
  are running python 2.7.6 or greater.  Python 2.7.6 is the first that supports
  unicode arguments to `struct.pack()`, which thrift requires.  Python 2.7.5 can
  import the thrift modules, but throws errors when trying to run them.

Reviewed By: bolinfest

Differential Revision: D4935279

fbshipit-source-id: 9af81736124c55476a5eb5beba9474a4371a639b
2017-04-24 11:14:34 -07:00
Adam Simpkins
ce0ce6fa4e move eden/fs/cli to eden/cli
Summary:
Move the code for the command-line tool up one directory, out of eden/fs.
This better separates the code so that eden/fs contains code for the edenfs
daemon, while eden/cli contains code for the command line tool.

Reviewed By: bolinfest

Differential Revision: D4888633

fbshipit-source-id: 5041e292c5353d05122eefe5db3257289e31239a
2017-04-14 11:39:01 -07:00
Adam Simpkins
4bb5948640 fix an invalid memory access in the checkout code
Summary:
Fix a subtle crash during checkout when handling newly added entries that
already exist in the working directory: CheckoutAction passed the entry name to
checkoutUpdateEntry() as a PathComponentPiece.  However, this
PathComponentPiece could refer to the entry name owned by newScmEntry_, and it
also passed newScmEntry_ into checkoutUpdateEntry() as an rvalue reference.
As a result, if the string data was stored invalidated by the move the name
would no longer be valid when checkoutUpdateEntry() tried to use it.

This bug is triggered by doing an "hg update --clean", where a file added in
the destination commit already exists on disk, and has an entry name of 23
characters or less.  (The 23 character limit is fbstring's upper bound on
small string optimizations, where it will store the string data inline in the
object, causing it to be invalidated on move.)

This also fixes a crash in a VLOG() statement when the verbose log level for
TreeInode.cpp was set to 4 or greater.

Reviewed By: bolinfest

Differential Revision: D4882544

fbshipit-source-id: 917ede6eeae2224aaa0724b8b30324f3c3a5c924
2017-04-13 17:34:38 -07:00
Adam Simpkins
8fcaefe6b2 implement hg reset
Summary:
Update the hg extension to implement dirstate.rebuild().  This is necessary for
the `hg reset` command.  This also now implements dirstate.setparents() for
cases when there is only one parent.

Reviewed By: wez

Differential Revision: D4823780

fbshipit-source-id: 802de006e03860995095dc3af17acb2eb05f4e8b
2017-04-06 17:50:43 -07:00
Wez Furlong
acb6539f87 add .eden/client symlink
Summary:
The intent is to provide a way to locate the SNAPSHOT file
for tools that want to have a very fast way to figure out the commit
id without making any RPCs or subprocess invocations.

Reviewed By: simpkins

Differential Revision: D4824176

fbshipit-source-id: 5adca225d9984146852dad1e83de0d903848c1e5
2017-04-06 13:20:02 -07:00
Adam Simpkins
d6f7a2f91a report timestamps on non-materialized files
Summary:
This diff fixes FileData::stat() so that we report reasonable timestamp values
on non-materialized files, rather than always leaving them as 0.  We set the
timestamps to the time that we created the FileInode.  This ensures that
timestamps are updated correctly when files are modified by a checkout
operation.

Note that for materialized files the code reports the timestamp of the overlay
file.  This diff does not modify that behavior.  However, this behavior is
incorrect, as the overlay file timestamps are not updated by a FUSE client
opening, modifying, then closing a file (since we keep the underyling overlay
file handle open, and don't close it).

In the future we'll need to implement our own tracking of atime, mtime, and
ctime values.  We should probably store these in a header inside the overlay
file.  For now, this diff is a stop-gap measure that ensures we at least update
non-materialized file timestamps correctly on checkouts.

Reviewed By: bolinfest

Differential Revision: D4765632

fbshipit-source-id: 478da6441e213cdfe830f1c5129212182ce4eeb0
2017-04-03 15:50:32 -07:00
Adam Simpkins
78b42e4208 make vlog settings controllable on a per-test basis
Summary:
Update the integration test code to allow TestCase classes to control the
--vmodule settings used when starting the edenfs daemon.

D3851805 initially set the vlog level for RequestData to 5 to help debug
issues.  However, this log level doesn't really make sense for most of the
mercurial integration tests: most mercurial commands checks for the existence
of lots of files, causing the logs to be filled with useless ENOENT messages
when the RequestData log level is this high.

Reviewed By: wez

Differential Revision: D4814402

fbshipit-source-id: 1127c0a25b656ea1a710ca54a59a9407d66a5659
2017-03-31 18:21:44 -07:00
Adam Simpkins
879fabdcc8 add an assert_status() method to the hg integration tests
Summary:
Add an assert_status() method to the hg integration tests that runs "hg
status", parses the output, then compares it to expected results.

Reviewed By: wez

Differential Revision: D4814422

fbshipit-source-id: 24ebdc2e0239c4833953c31e5786cc320bcd9d62
2017-03-31 18:21:44 -07:00
Adam Simpkins
8751f1b702 print stderr when an hg command fails in integration tests
Summary:
Update HgRepository.hg() to catch subprocess.CalledProcessError exceptions and
translate them into subclass that returns a more helpful message.

The CalledProcessError includes the command stderr as a member variable, but
unfortunately does not include this in the output from `__str__()`.  The stderr
output is usually important for helping debug test failures.

Reviewed By: wez

Differential Revision: D4814401

fbshipit-source-id: 4f010bb3ce33833c55d18768997d2d1bb6a001b8
2017-03-31 18:21:44 -07:00
Adam Simpkins
82cc79b694 update hg_import_helper.py to invalidate the repo if an error occurs
Summary:
The hg_import_helper script that eden uses to import data from mercurial keeps
a long-lived repository object open.  This caches some data about the
repository, and if new commits are added after it was created, it can fail to
see them.

This updates hg_import_helper.py to catch errors that occur when trying to use
the repository objects.  The code will invalidate the repository object and
then retry the operation once, in the hopes that it will now succeed after
invalidation.

Reviewed By: bolinfest

Differential Revision: D4752659

fbshipit-source-id: 1c75c84766d6bbda0710882a338eaa09e0cb0030
2017-03-31 14:07:14 -07:00
Adam Simpkins
080134ea84 ignore ENOENT errors when invalidating FUSE inode entries
Summary:
The kernel can return ENOENT in response to invalidation notification if we
have never told the kernel about the inode in question.  This resulted in
spurious errors during checkout when updating files that were loaded internally
by edenfs rather than via FUSE call.  For instance, this was commonly triggered
by .gitignore files, which eden loads on its own to perform ignore processing.

Reviewed By: bolinfest

Differential Revision: D4752630

fbshipit-source-id: d4e092643a8d33cf33709f7e3664289f167ac093
2017-03-31 14:07:14 -07:00
Adam Simpkins
a18e042e1c tweak the hg integration test base class
Summary:
I found it rather awkward in HgExtensionTestBase that self.repo is not actually
the repository being tested.  It was instead the repostiory used as the backing
store for the mercurial data, and self.repo_for_mount was the repository being
tested.

This diff renames the two repository classes, so that self.backing_repo is now
the backing store repository, and self.repo is the repository being tested.

In order to do this I changed HgExtensionTestBase to derive directly from
EdenTestCase.  Previously it derived from EdenHgTest, and was letting
EdenHgTest set up self.repo.  It seemed more understandable to avoid deriving
EdenHgTest now since self.repo is not the repository that needs to be set up
initially.

Reviewed By: bolinfest

Differential Revision: D4752631

fbshipit-source-id: d8b542b0ecead66b965af1a582085345e28b2908
2017-03-31 11:39:48 -07:00
Adam Simpkins
49adc9c96b fix the behavior of "hg update --clean ."
Summary:
Previously the eden hg extension short-circuited the checkout operation if the
destination commit was the same as the one currently checked out.  This was
incorrect if --clean was specified, since we do need to reset the working
directory state in this case.

This updates the extension code to always make the thrift checkout() call when
doing a force checkout.

This also avoids calling applyupdates() to resolve conflicts when force=True.
When doing a force checkout, eden reports files with conflicts that it
overwrote, but these do not need to be resolved by mercurial.

Additionally, this also updates a couple other APIs that have recently been
changed in upstream mercurial: merge.update() now takes an updatecheck
argument, and repo.join() should now be written as repo.vfs.join().

Reviewed By: bolinfest

Differential Revision: D4752510

fbshipit-source-id: e1ee92d086315e35a1378f674e668876a667c0ce
2017-03-31 11:39:48 -07:00
Wez Furlong
7c6f9608c2 add basic mknod support
Summary:
this is the bare minimum to support creating unix domain sockets.

We only support using mknod to create a unix socket; other uses will yield an error.

I've added an rdev field as a sibling of the existing mode field that we track,
as that is the additional parameter that we need to track as part of the
special file node.

Special file nodes are tracked in the overlay as empty files.

Reviewed By: bolinfest

Differential Revision: D4774099

fbshipit-source-id: 0824b7e509063faa8bede7aff82a7c51930c4f83
2017-03-30 23:53:05 -07:00
Wez Furlong
4235784907 add .eden "magic" dir
Summary:
It's not really magic because we don't have a virtual directory
inode base any more.  Instead, we mkdir and populate it at mount time.

What is slightly magical about it is that we give it some special powers:

* We know the inode number of the eden dir and prevent unlink operations
  on it or inside it.
* The .eden dir is present in the contents of the root inode and will
  show up when that directory is `readdir`'d
* When resolving a child of a TreeInode by name, we know to return the
  magic `.eden` inode number.  This means that it is possible to `stat`
  and consume the `.eden` directory from any directory inside the eden
  mount, even though it won't show up in `readdir` for those child dirs.

The contents of the `.eden` dir are:

* `socket` - a symlink back to the unix domain socket that our thrift
  server is listening on.  This means that it is a simple
  `readlink(".eden/socket")` operation to discover both whether a directory
  is part of an eden mount and how to talk to the server.

* `root` - a symlink back to the root of this eden mount.  This allows
  using `readlink(".eden/root")` as a simple 1-step operation to find
  the root of an eden mount, and avoids needing to walk up directory
  by directory as is the common pattern for locating `.hg` or `.git`
  dirs.

Reviewed By: simpkins

Differential Revision: D4637285

fbshipit-source-id: 0eabf98b29144acccef5c83bd367493399dc55bb
2017-03-24 23:07:42 -07:00
Adam Simpkins
38bbe966ab fix a bug in bind mount shutdown
Summary:
The privhelper code was erasing elements from a std::unordered_map before it
was done using the iterator pointing to that element.  This causes memory
corruption issues.

Between this and some of my other recent unmount fixes (D4548030, D4547938)
this makes the bind-mount-related integration tests work.

Reviewed By: bolinfest

Differential Revision: D4727850

fbshipit-source-id: 6d1fda3f89cb91c89d0020921b1805fc10e65785
2017-03-20 22:11:19 -07:00
Wez Furlong
82612ba4d6 remove getMaterializedEntries thrift API
Summary:
I originally added this to facilitate `hg status` but we didn't end up
using it.  Rather than keeping it around and updating the tests in the `.eden`
dir diffs that follow, let's just remove it.

Reviewed By: simpkins

Differential Revision: D4610768

fbshipit-source-id: 158c0207f88980e86aeeddf75e6fd49763d2a402
2017-03-01 08:19:29 -08:00
Adam Simpkins
8884b46b3f move integration tests to eden/integration
Summary:
Move the integration tests from eden/fs/integration up one directory, to
eden/integration.

The main benefit is that this makes it easy to run just the edenfs unit tests
by running "buck test eden/fs/...".  These unit tests complete much more
quickly than the full set of integration tests, providing a faster test suite
to re-run repeatedly during development.  The integration tests can be run with
"buck test eden/integration/...", and the full set of tests can still be run
with "buck test eden/..."

Reviewed By: wez

Differential Revision: D4490247

fbshipit-source-id: 5ceb5a19526f56e1cb926f352fa30ad2f1212c05
2017-01-31 14:41:14 -08:00