Commit Graph

187 Commits

Author SHA1 Message Date
Adam Simpkins
19ce8cdcd8 update HgRepository.hg() to always return a string
Summary:
Drop the `stdout` and `stderr` arguments, so that this method always return a
string.  Change callers that were previously calling this method with
`stderr=None` to use the `run_hg()` method instead of `hg()`.  `run_hg()`
returns a `subprocess.CompletedProcess` object.

This change simplifies the python type checking, and fixes several existing
type checking errors in the code.  Even though most call sites could be
guaranteed that this function would return a `str`, the type checker wasn't
smart enough to tell that the return type would be fixed based on the argument
values, and so it assumed the result always needed to be checked for `None`.

This also updates the `GitRepository.git()` method in a similar fashion.
However, that was a simpler change since it already returned a `str` in all
cases.

Reviewed By: strager

Differential Revision: D13078095

fbshipit-source-id: a8def2a33edc865ac40279bbcb3ada4dade68374
2018-11-16 19:47:48 -08:00
Adam Simpkins
1b389f52b4 restructure storage_engine_test to make the type checkers happy
Summary:
Restructure storage_engine_test so that the base class derives from
EdenTestCase.

Reviewed By: strager

Differential Revision: D13051096

fbshipit-source-id: e89c4b56e361460b2457d1c2e6a22727a25d7646
2018-11-16 19:47:47 -08:00
Adam Simpkins
a0411b8a51 add member type annotations to make pyre happy
Summary:
Add type annotations for class member variables.  The pyre type checker has
some limited automatic type detection for member variables set in
`__init__()`, but in general it expects member variables to be explicitly
declared at the top-level of the class.

Reviewed By: strager

Differential Revision: D13051092

fbshipit-source-id: 080259ab3f422ffae2b908ed610062237105ccbe
2018-11-14 13:03:09 -08:00
Adam Simpkins
43fb04699f re-enable test_hg_clone_non_eden_repo_within_eden_repo for treemanifest
Summary:
The `test_hg_clone_non_eden_repo_within_eden_repo()` test had been disabled
when running with treemanifest enabled since the treemanifest code did not
correctly detect which repositories should actually use treemanifest.

This issue appears to have since been fixed in the treemanifest code, so this
test passes now.

Reviewed By: quark-zju

Differential Revision: D12927801

fbshipit-source-id: b9e2e041f7eab5e24007888e2dba142e1f0b2251
2018-11-06 18:08:09 -08:00
Adam Simpkins
d08f63df8f tests: change HgRepository.status() to return a dictionary
Summary:
Update the HgRepository.status() function in the integration tests to return
the status information as a dictionary instead of a string.  Only one test was
still using the old string API.

Reviewed By: chadaustin

Differential Revision: D10503168

fbshipit-source-id: 574e4438d23bf6612a70ae5ae3174db3d464d198
2018-10-23 13:39:35 -07:00
Adam Simpkins
daa882b16f move code to find the post clone hook to find_executables.py
Summary:
Update the eden+hg integration test code so that it uses find_executables.py
to find the post clone hook.  Previously it was still using its own one-off
helper function for this.

Moving this code into find_executables.py will make it possible have the
snapshot tool also find the post clone hook.

Reviewed By: chadaustin

Differential Revision: D10503169

fbshipit-source-id: d59e3e517d19e80e0d0c997be09865519b80d34e
2018-10-23 13:39:35 -07:00
Chad Austin
2a9e1e2f29 remove fallback for correcting empty files
Summary:
Now that the import bug has been fixed for some time, it's likely few
people have cached empty files. And if they do, `eden gc; eden
restart` is a fine workaround.

On the other hand, reimporting empty files gums up the importer, and
I've seen several people recently complaining about performance that
could be partially attributed to the fact that their Eden needed to
verify empty files. simpkins has talked about adding a bit to the
cached blob to determine whether it needs reimporting or not, but it's
probably going to take a while for anyone to implement that, and this
reverification logic is hurting people today.

Reviewed By: strager

Differential Revision: D10456519

fbshipit-source-id: 657bc377ee16ce93494075bde4388aed59dceecf
2018-10-22 20:27:26 -07:00
Phil Cohen
16c69525bd filemerge: add number of textual conflicts in each file to error message
Summary:
When you get an error, let's print the number of conflicts in each file. This will give the user some sense of how much work they have to do.

The code change is entirely in `filemerge.py`, and `tests/test-merge-conflict-count.t` adds a new test.

Reviewed By: quark-zju

Differential Revision: D9815243

fbshipit-source-id: 1b73a1db293902ac7242997a7d6ae09478344068
2018-10-22 20:27:26 -07:00
Mark Thomas
a2dd1e594e sparse: add sparse show
Summary:
Add a new sparse show command, which shows the current sparse configuration.
Use common templating, including colors, between sparse show and sparse list.

Reviewed By: phillco

Differential Revision: D10446705

fbshipit-source-id: acb539f80f625945716758c785703b46a81aa6f1
2018-10-22 20:27:26 -07:00
Mark Thomas
870f1cb843 help: categorize sparse and cloud subcommands
Reviewed By: phillco

Differential Revision: D10446706

fbshipit-source-id: ef6dad180930f4203e249583fbf0e913cbf01b14
2018-10-22 20:27:26 -07:00
Mark Thomas
47beebefea tidy up command usage
Summary: Some commands in extensions have incorrectly defined usage.  Correct these.

Reviewed By: phillco

Differential Revision: D10446707

fbshipit-source-id: e606314892a31fa09f4a655042d66d87929fc2ec
2018-10-22 20:27:26 -07:00
Kostia Balytskyi
09c68ebb4a fix sparse tests
Summary: D10335083 renamed `fbsparse` into `sparse` and I forgot to udpate this test.

Reviewed By: simpkins

Differential Revision: D10423192

fbshipit-source-id: a547c403a9461d6923ecfecc931df82bb7887597
2018-10-22 20:27:25 -07:00
Phil Cohen
900e52912d tweakdefaults: port allowbranches to core
Summary:
Note that this is more aggressive than the tweakdefaults version; it doesn't allow named branches to be created or listed at all with the config set. As we want to delete named branches entirely, this seems reasonable.

I ported the `--new` flag, but made it optional, to make it easier to support both the core and tweakdefaults callers.

Saurabh Singh and I chatted with the releng team (Craig).

**tldr:** we can remove named branches from the UI but should keep the innards for now

Today the releng team doesn't make any branches (#continuous), but in the near-term future, they might either:

- create remote bookmarks in hg (similar to fbsource)
- create branches in SVN (which are then synced as hg branches by hgsubversion)
  - We'll leave all the hgsubversion code to do this alone, just remove user-facing entry points
  - They'll also need to be able to query for a commit's branch (`hg log -T '{branch}'` will still work)
  - Once www is on hg, this possibility goes away and we can kill named branches for good

Reviewed By: ikostia

Differential Revision: D10401485

fbshipit-source-id: 6f2f3ae28af249bae1fdf782eb14fe7bfc472bb7
2018-10-22 20:27:25 -07:00
Mark Thomas
91f61a5c6a absorb: display affected commits and prompt user to confirm
Summary:
Update hg absorb to use the templater and display the first line of the
description of the affected commits.

Also change the default behaviour to display the changes and prompt the user to
confirm that absorb is going to do what they want.  Use the new `-a` option to
get the old behaviour.

Reviewed By: quark-zju

Differential Revision: D10366510

fbshipit-source-id: 23df2e5ab2d21a0805a16e7118686a29634d2918
2018-10-13 01:02:05 -07:00
Saurabh Singh
ce30b1e029 amend: replace with the fbamend extension
Summary:
The functionality we care about is provided by the `fbamend`
extension. Therefore, lets replace the `amend` extension with the `fbamend`
extension.

Reviewed By: farnz

Differential Revision: D10320739

fbshipit-source-id: 5700d39f488777fcc4033f60ce0a51cda15ef2ad
2018-10-11 06:59:05 -07:00
Chad Austin
d5c1e599b8 remove support for non-toml configs
Summary:
We've been shipping our RPMs with toml config support turned on for a
while now. Remove support for the old config file format. Continue to
ship the old format configurations. We'll remove those in a later
diff.

Reviewed By: strager

Differential Revision: D10020958

fbshipit-source-id: 11c2ca3b5da086b142042496a2814699880c4f81
2018-10-10 12:49:16 -07:00
Saurabh Singh
72e21b6f57 inhibit: remove the extension
Summary:
The logic we care about has been moved to core so we can delete the
`inhibit` extension.

Reviewed By: markbt

Differential Revision: D10276448

fbshipit-source-id: 44cabe5d561344cf8f196127ce52491e4654b598
2018-10-10 10:01:33 -07:00
Adam Simpkins
13ef80fe54 fix dirstate.walk()'s handling of ignored symlinks
Summary:
Fix the code in `eden_dirstate._call_match_callbacks()` to correctly match
ignored symlinks that are explicitly listed in the matcher's file list.

It looks like this was just a mistake in the original code on my part: I
intended to check both `S_ISREG()` and `S_ISLNK()` but instead incorrectly put
the check for `S_ISREG()` twice.

Reviewed By: wez

Differential Revision: D9476524

fbshipit-source-id: 67e0fa7c2fbaac97598a8e2d028c9ef0999ed88a
2018-08-23 15:07:02 -07:00
Adam Simpkins
a88763ae96 re-verify blob contents for empty blobs loaded from the LocalStore
Summary:
When we load an empty blob from the LocalStore double check with the
BackingStore to confirm that it should actually be empty.

We have seen multiple instances of files being incorrectly imported as empty.
So far this error has always been fixed by a re-import.  We still haven't
tracked down the root cause, but this change should help workaround the issue
by ensuring that we double check the file contents before returning the data.

Reviewed By: chadaustin

Differential Revision: D9476522

fbshipit-source-id: 6d57cf15c42736ecbcb106a731259b77db06d8f1
2018-08-23 14:22:58 -07:00
Chad Austin
164af00a35 run some integration tests across all three LocalStore implementations
Summary:
To prevent the regression fixed by D8323051, make sure we run at least a couple
integration tests with the RocksDB LocalStore implementation.

Reviewed By: wez

Differential Revision: D8408390

fbshipit-source-id: 8fab4041ae39915d8be80e42814aab375c4acdda
2018-08-15 10:22:02 -07:00
Adam Simpkins
a5f811f2eb normalize "hg journal" by stripping off extra profiling flags
Summary:
The eden integration tests currently run hg through the telemetry wrapper.

This wrapper script adds profiling arguments to a percentage of the commands
it runs.  The extra arguments that it adds are not entirely transparent to the
user, and are unfortunately reported in the output of "hg journal".

This was causing occasional test failures in the Eden integration tests that
check the output of "hg journal".

This diff changes the Eden code that checks the journal output to try to strip
off these flags before checking the command output.

Reviewed By: wez

Differential Revision: D8631180

fbshipit-source-id: 004d3bcce291ffc2208f85a29cb34a921f16fdde
2018-06-26 16:54:05 -07:00
Wez Furlong
fdd530c007 ensure .hg has read and search permissions for group/other
Summary:
mkdtemp deliberately removes these bits and doesn't respect
the umask.  We perhaps should respect the umask ourselves here.

Reviewed By: chadaustin

Differential Revision: D8335539

fbshipit-source-id: 98b8f3e2db17c65904b12627e52966d5f6b729ef
2018-06-11 22:33:11 -07:00
Wez Furlong
2c830c5e57 improve sparse extension error handling when used with eden
Summary:
Some folks might attempt to load this extension despite it
not being useful with eden.  The extension knows not to hook in to
the repo when eden is in used, but the various commands to manipulate
the sparse config do not.

This diff adds a little helper function to show a more reasonable
error message than an ugly stack trace when attempting to use them.

Reviewed By: simpkins

Differential Revision: D8328229

fbshipit-source-id: d7cc4c7047b11f8da96e26ab70bd522b52546151
2018-06-08 14:55:40 -07:00
Chad Austin
b9f6bf1c14 add clear_local_caches debug command to cli
Summary:
This adds a debug command to blow away all RocksDB information that
can be reproduced from Mercurial. We will use it to help an Eden user
recover from a corrupted blob.

Reviewed By: bolinfest

Differential Revision: D8108649

fbshipit-source-id: 056dec19d51b9e430b3c2a249747b26830cfc875
2018-05-31 11:23:21 -07:00
Adam Simpkins
b9e09a508f add a new status test for restoring a files contents
Summary:
Add a new integration test that modifies a file, commits it, then restores it
to its original contents and creates a new commit with the revert.

This test isn't really exercising any Eden-specific behavior.  The only reason
it is is interesting is that this happened to trigger a bug in the underlying
mercurial cdatapack code that caused the test to fail when run in treemanifest
mode.  The mercurial cdatapack bug is fixed in D8131020

Reviewed By: quark-zju

Differential Revision: D7826284

fbshipit-source-id: 32da2db04452d5df0527f6be00b0c4c1a56c900b
2018-05-30 18:52:52 -07:00
Lukasz Langa
deee232d74 Upgrade to 18.5b1
Summary: Mostly empty lines removed and added.  A few bugfixes on excessive line splitting.

Reviewed By: cooperlees

Differential Revision: D8198776

fbshipit-source-id: 4361faf4a2b9347d57fb6e1342c494575f2beb67
2018-05-30 01:11:47 -07:00
Lukasz Langa
bf7f0a79b8 Reformat already opted in directories with Black Beta @allow-large-files
Summary:
This is stacked on top of Black 18.5b0.

allow-large-files

Reviewed By: carljm

Differential Revision: D8061834

fbshipit-source-id: 92e3645e159b60d77cf7e0bec64a8262ca4e88c2
2018-05-18 23:07:24 -07:00
Adam Simpkins
c8e69b61fb only call resetParentCommits() on dirstate write
Summary:
This updates the Eden mercurial extension to no longer invoke the Eden
`resetParentCommits()` thrift call when `setparents()` is called on the
dirstate map.  Instead we now defer the call to `resetParentCommits()` until
`write()` is called to write the dirstate data to disk.

Informing edenfs of the parent change as soon as `setparents()` was called was
problematic, as this made edenfs reflect the change before the transaction was
committed.  Some mercurial commands, notably `hg status` also call
`setparents()` on the dirstate but never write this back to disk at all.  This
is problematic since `hg status` calls `setparents()` without holding any
mercurial locks.  As a result it may call `setparents()` with the "wrong"
parent if another mercurial process is running and is in the middle of a
transaction.

Reviewed By: bolinfest, chadaustin

Differential Revision: D7980375

fbshipit-source-id: 4f5e4391fd291d4ea5fc93bb9d49ed0380fc1721
2018-05-14 12:18:04 -07:00
Wez Furlong
c83849e5af enable Black python formatting and apply to eden
Summary: No functional changes

Reviewed By: simpkins

Differential Revision: D7945989

fbshipit-source-id: e267e6134d87570427b3fdf5974006dce5774113
2018-05-09 21:37:07 -07:00
Adam Simpkins
f187baef89 fix bug incorrectly reporting a file as modified
Summary:
When comparing two source control blob hashes, identical hashes can be assumed
to mean that the file contents are equal.  However, differing hashes does not
necessarily mean that the file contents differ.  In particular, mercurial
hashes history metadata in addition to the file contents when computing the
blob hash.

This updates Eden to always compare the file contents when the source control
blob hashes differ, rather than assuming that the file contents are different.

Reviewed By: wez

Differential Revision: D7825900

fbshipit-source-id: e611124a66cdd5c44589f20d1d4665a603286530
2018-04-30 22:10:01 -07:00
Adam Simpkins
b2251c9f7a change mercurial to look up the correct Eden mount path
Summary:
Update the Eden mercurial extension to read the `.eden/root` symlink to
determine what Eden thinks the mount path is.  This might be different from
what directory mercurial thinks it is in if a parent directory of the Eden
mount has been bind-mounted to an alternate location.

Maybe in the future we should update thrift clients to pass in the client ID
(currently readable via `.eden/client`) rather than the mount path.  That would
make it less likely for clients to accidentally forget to read `.eden/root` and
pass in the wrong mount path.

Reviewed By: wez

Differential Revision: D7705655

fbshipit-source-id: 7bd1e8013b99a52ff06dd45f63d6669b66bdf577
2018-04-24 13:11:35 -07:00
Adam Simpkins
cc484caf6c remove a stale comment from some of the test code
Summary: I forgot to remove this comment when I fixed the test in D7565503.

Reviewed By: chadaustin

Differential Revision: D7716106

fbshipit-source-id: 58acd71ac96560f2cc454b3a7da75bb61e6e71c7
2018-04-20 15:38:11 -07:00
Michael Bolin
b07ddbc2b3 Fix flake8 warnings in eden/integration folder.
Summary:
These were making some noise in the Nuclide diagnostics pane.

In the course of making these changes, I ended up reformatting
some files, as well. Perhaps we should flag flake8 and autoformat
violations more aggressively in CI now?

Reviewed By: chadaustin

Differential Revision: D7658030

fbshipit-source-id: b52a29b1cc242967f868dcc8ee46ec1bb9bdfbee
2018-04-18 12:42:33 -07:00
Adam Simpkins
2136d20933 improve the behavior of the noconflict check in hg update
Summary:
Fix a couple issues when when performing an `hg update` with updatecheck set to
`noconflict` and when conflicts are detected:

- Report the files with conflicts.  The conflicts may be non-obvious to the
  user if the conflicts are in ignored files that are not tracked in the
  current commit but are present in the destination commit.
- Do not invoke the preupdate hook or create the .hg/updatestate file until
  after we have performed the conflict check.  Otherwise the repository will be
  marked as in the middle of an unfinished update even though we never
  attempted the update itself.

Reviewed By: quark-zju

Differential Revision: D7581782

fbshipit-source-id: e290ebd6a27f228a805bea40aabb3f3cc633cc7d
2018-04-13 16:19:22 -07:00
Adam Simpkins
62e6399a85 make assert_status() check for unfinished update/rebase/etc
Summary:
Update the assert_status() function in the integration tests so that it also
checks for unfinished update/rebase/graft/etc operations.  We unfortunately
have to manually check for the presence of these files ourselves, since
`hg status` provides no mechanism to report this data when `HGPLAIN` is set.

Reviewed By: quark-zju

Differential Revision: D7581781

fbshipit-source-id: 230234e5b8ce28cd3569cdacced686fed2a9dd32
2018-04-13 16:19:22 -07:00
Adam Simpkins
3905f37258 re-implement dirstate.walk() and dirstate.status()
Summary:
This rewrites the `walk()` and `status()` methods in the `eden_dirstate` class.

The changes to the `status()` function should not have any major behavior
changes: this primarily moves logic out of `EdenThriftClient.getStatus()` and
into `dirstate.status()`

The changes to the `walk()` function do fix a number of bugs.  We now implement
walk by combining Eden's `getScmStatus()` results with the dirstate's
non-normal files and the current parent commit's manifest.  All of the glob
matching is done purely in python now.  This fixes some cases where Eden's glob
handling behavior did not support some glob patterns allowed by mercurial.  The
Eden glob() call also always returned ignored files and hidden files (such as
everything inside the `.hg` directory).  I believe this behavior is desired by
watchman, but it caused problems for the `walk()` code.

Reviewed By: chadaustin

Differential Revision: D7565503

fbshipit-source-id: deb91b4772501e8fbdba56e5c099a72d55fb61ee
2018-04-13 16:19:22 -07:00
Adam Simpkins
81a645c21c fix dirstate.status() to invoke match callback functions correctly
Summary:
Update `eden_dirstate.status()` to invoke the `match.bad()` and
`match.explicitdir()` appropriately when any of the explicitly specified match
paths either don't exist or refer to directories.

Reviewed By: chadaustin

Differential Revision: D7556850

fbshipit-source-id: 2e8d88ff0e3c09cea2f3ef5b18c9a214231815ab
2018-04-13 16:19:22 -07:00
Adam Simpkins
f96f244cc0 fix the behavior of dirstate.matches() in Eden
Summary:
Implement `dirstate.matches()` in Eden by calling `matches()` on the parent
commit's context, and then updating that result with the list of non-normal
files from Eden's dirstate.

Reviewed By: chadaustin

Differential Revision: D7556847

fbshipit-source-id: 34e78c0c125cd1b184a7773a149edafdca34fee1
2018-04-13 15:07:40 -07:00
Adam Simpkins
ac03fb0a96 add integration tests for hg journal
Summary:
Add integration tests to check the behavior of the `hg journal` command in an
Eden checkout.

Reviewed By: wez

Differential Revision: D7512339

fbshipit-source-id: 9a83a1752835bc6ded9f0c3b2cb29ec24a6db631
2018-04-13 15:07:40 -07:00
Adam Simpkins
8ac9c65807 use FB-specific system hgrc configs in integration tests
Summary:
When running the integration tests, set HGRCPATH to point to the configs found
at `scm/hg/fb/staticfiles/etc/mercurial`

This does affect mercurial's behavior in some cases, and therefore required
changes to some of our tests:
- `hg diff` output now uses an output style similar to `git diff`
- merge conflict markers include some extra commit metadata

Reviewed By: wez

Differential Revision: D7512321

fbshipit-source-id: 6141605147797ea8b13fdb5a06f67beaeff6afe7
2018-04-13 15:07:40 -07:00
Adam Simpkins
d3f0166b6c add create_git_repo() and create_hg_repo() methods to EdenTestCase
Summary:
This slightly refactors the way that EdenTestCase and EdenRepoTest initialize
repositories.  This removes the `create_repo()` method from EdenTestCase and
replaces it with separate `create_hg_repo()` and `create_git_repo()` methods.

The `EdenRepoTest` subclasses now provide alternate `create_repo()`
implementations instead of `get_repo_class()`.

This cleans up the code a bit, since `create_repo()` no longer takes different
arguments based on what type of repository is being created.  This also will
make it easier in upcoming diffs to further customize the logic that occurs in
`create_hg_repo()`.

Reviewed By: chadaustin

Differential Revision: D7512320

fbshipit-source-id: d268b0ac0ffb33f3dfd34f2dd1917d57033c81aa
2018-04-13 15:07:40 -07:00
Adam Simpkins
aa3009b2b8 clean up the find_executables module in the integration tests
Summary:
Refactor the find_executables module to only look up executables when they are
needed, instead of performing all look-ups immediately even if the test in
question may not need all of the binaries.  Also add a _find_exe() helper
function to eliminate some code duplication.

Reviewed By: ryanmce

Differential Revision: D7512038

fbshipit-source-id: fdfb8ec70b3f6292603826b3fb22c01dbd1f0d72
2018-04-06 12:36:52 -07:00
Adam Simpkins
398a824aac add type annotations to more of the integration tests
Summary:
We already had type annotations on most of the `hg` integration tests.  This
adds them for the top-level (non-source-control-specific) tests.

typeseverywhere

Reviewed By: wez

Differential Revision: D7459281

fbshipit-source-id: 41266b232ded510d6b63dd3e62c272a0cd6a0e1a
2018-04-04 17:55:11 -07:00
Adam Simpkins
05029df010 disable update_test.test_dir_locking on sandcastle
Summary:
This integration tests appears to be frequently hanging on sandcastle, and
leaving behind unkillable zombie processes.  I'm disabling it for now until we
can figure out why it hangs and fix it and/or make sandcastle smart enough to
be able to successfully tear down the FUSE mounts.

These stuck processes are likely stuck waiting on a kernel lock that can only
be released by using `/sys/fs/fuse/connections/DEVICE_ID/abort`

Reviewed By: wez

Differential Revision: D7440282

fbshipit-source-id: 70034fd65267eff18bf5a052cde8d11e0a66b46b
2018-03-29 20:13:16 -07:00
Adam Simpkins
086fc9ac30 send FUSE invalidation requests in a separate thread
Summary:
Update FuseChannel to send all invalidation requests in a separate thread.

This eliminates a deadlock that could previously occur during checkout
operations.  The invalidation requests would block until they could acquire the
kernel's inode lock on the inode in question.  However, the inode lock may
already be held by another thread attempting to perform an unlink() or rename()
call.  These FUSE unlink or rename operations would be blocked waiting on
Eden's mount point rename lock, which was acquired by the checkout operation.

Checkout operations now let the invalidations complete asynchronously, but we
wait for all invalidation operations to complete before indicating to our
caller that the checkout has succeeded.

Reviewed By: chadaustin, wez

Differential Revision: D7404971

fbshipit-source-id: 6fa20c00d054e210eb0258d247d083010557f210
2018-03-27 11:23:42 -07:00
Adam Simpkins
4b49e012ec move the tree generation utility function to a helper module
Summary:
The status_deadlock_test code has a small helper function used to create
directory trees in the test.  This moves that function into a helper module so
we can re-use it in other tests in the future.

Reviewed By: chadaustin

Differential Revision: D7407492

fbshipit-source-id: 257e5a2ce7543bb6cd218b412d165f0fac852970
2018-03-26 19:20:33 -07:00
Adam Simpkins
bf33c996c6 allow FUSE caching of negative lookup() responses
Summary:
Previously we returned an ENOENT error in response to a FUSE lookup() call for
a name that does not exist.  However, this does not allow FUSE to cache the
result, so we will continue to receive lookup() calls for this path in the
future.

This changes EdenDispatcher to return a successful response with an inode
number of 0 instead.  This tells the kernel that the name does not exist, but
allows the kernel to cache this negative lookup result (for as long as
specified in the entry_valid field in the response).

Reviewed By: wez

Differential Revision: D7076811

fbshipit-source-id: a2b9977e58d6b6eecb584699b9d93b5ad29ad5ad
2018-02-26 19:50:47 -08:00
Adam Simpkins
1f21fa3361 add an integration test for "hg pull"
Summary:
Add a test that exercises `hg pull`.  This confirms that eden can see new
commits created on the server after Eden and its hg_import_helper processes
have started.  This test gets run in flatmanifest, hybrid treemanifest, and
treeonly mode.

This currently performs pulls using a local peer repository rather than over
SSH.  This does exercise a different code path in mercurial than what typically
occurs in production.  In the future we should perhaps also add a test that
uses a fake SSH helper program to exercise mercurial's sshpeer code paths as
well.

Reviewed By: chadaustin

Differential Revision: D6993788

fbshipit-source-id: 40628c0b3faac0dc8622b605a29b084979b8c089
2018-02-21 18:57:58 -08:00
Chad Austin
7975e747b9 fix hg fold
Summary:
Whenever we tell Eden to change the working directory
parents, we need to make sure the appropriate objects are written to
disk.  This fixes hg fold in Eden.

Reviewed By: simpkins

Differential Revision: D7045299

fbshipit-source-id: cbd51be59cf943a843b77c2abe66a84b745bce22
2018-02-21 16:26:39 -08:00
Adam Simpkins
807430b754 update the integration tests to use hg.par's builtin eden extension
Summary:
Update the integration tests to avoid specifying an explicit path to the eden
extension.  This way they use the version that we now package into hg.par
during the build.

This avoids issues with hg not being able to find and load native .so libraries
from the eden extension.  Mercurial is able to find these libraries correctly
when they are packaged into hg.par (since the par start-up script sets
LD_LIBRARY_PATH to point to the par unpack directory).  When using eden from an
external directory mercurial was not able to find these libraries.

Reviewed By: chadaustin

Differential Revision: D7047245

fbshipit-source-id: d56bffa953c178949c866efec507298a1f40da8b
2018-02-21 15:24:49 -08:00
Adam Simpkins
21d2b6c46d Remove TARGETS files
Summary:
This removes the TARGETS files from the eden github repository.  The
open source buck build has been failing for several months, since buck
removed support for the thrift_library() rule.

I will potentially take a stab at adding CMake build support for Eden
at some point in the future.

Reviewed By: chadaustin

Differential Revision: D6893233

fbshipit-source-id: e6023094a807cf481ac49998c6f21b213be6c288
2018-02-20 19:57:45 -08:00
Adam Simpkins
d922d6cec5 add a flag to disable flatmanifest fallback
Summary:
Add a command line flag to control whether or not Eden should ever try falling
back to import tree data using flatmanifest if an error occurs trying to import
it directly via treemanifest, in repositories that support treemanifest.

This is particularly useful for tests, where we usually do not want to fall
back to flatmanifest import if an error occurs during treemanifest import.  The
fallback can otherwise mask real issues that should trigger test failures.

This is probably also a good thing to have in general.  Supporting
flatmanifest+treemanifest data in a single Eden repository has some unfortunate
problems today: we compute hashes differently for flatmanifest trees vs
treemanifest trees.  As a result, we can end up with identical trees that have
different hashes.  This can result in unfortunate performance consequences in
some cases where Eden assumes it must scan a directory for differences if the
hashes are different.

I have left flatmanifest import enabled by default for now, but we may want to
disable it by default in the future.  I would be more inclined to disable it by
default if we did added a thrift method to explicitly re-enable it (or to
import a single commit using flatmanifest), so that users could work around
this setting if necessary without having to fully restart edenfs.

Reviewed By: wez

Differential Revision: D6993791

fbshipit-source-id: 6e091a426cf1e7c973df5a641d2f8a1101011346
2018-02-15 22:11:38 -08:00
Adam Simpkins
9dbdeaa4f4 add the ability to run some integration tests in treeonly mode
Summary:
Update the `hg_test` decorator to accept additional parameters specifying the
list of test modes.  e.g., `hg_test('Treemanifest')` asks to only run the test
in the Treemanifest configuration.  With no arguments tests are still run with
both the Flatmanifest and Treemanifest configurations by default.

This also enables the TreeOnly configuration mode, which appears to work now.
(It was previously disabled since `hg init` would fail in treeonly
repositories.)

This new changes allows tests to explicitly opt-in to running in `TreeOnly`
mode.

Reviewed By: wez

Differential Revision: D6993789

fbshipit-source-id: 9ee51318d0f661038fe29f246b2b14eebbb1c3d9
2018-02-15 11:41:28 -08:00
Chad Austin
144a47d1ae fix status/diff when symlinks change
Summary:
isSameAs calls getSha1 which was failing on symlinks.  The
original concern was that asking for the SHA-1 of a symlink is
ambiguous: do you want the hash of the symlink or the target?  But we
already check for whether you are requesting the SHA-1 of a symlink in
EdenServiceHandler, so it's redundant and incorrect to check in
FileInode too.

Reviewed By: simpkins

Differential Revision: D6847489

fbshipit-source-id: 13966da06bcde75c5c568e09fef14e735de47cfb
2018-01-30 13:35:49 -08:00
Adam Simpkins
27ef593f70 add type annotations to all functions in commit_test.py
Summary: Add type annotations to the test functions in this file.

Reviewed By: chadaustin

Differential Revision: D6789863

fbshipit-source-id: d298a388b5ea03fcf25047937e747051c8fa682b
2018-01-23 20:50:29 -08:00
Adam Simpkins
4abb5f6720 add integration tests for running "hg commit <path>"
Summary:
Test running `hg commit` with a path argument.  This currently fails if the
path refers to a directory rather than a file.

Reviewed By: chadaustin

Differential Revision: D6789862

fbshipit-source-id: 054ee26ea0fdb843cc12d5e8ac3c3d34b9308017
2018-01-23 20:50:29 -08:00
Adam Simpkins
78262a0cf6 add some additional checks to the hg update --merge test
Summary:
Update the `hg update --merge` to test marking the change resolved and
committing the result.

Reviewed By: wez

Differential Revision: D6757299

fbshipit-source-id: c147e6ff8b63e364f96d2ce8a2094dd809d8da7d
2018-01-19 15:51:47 -08:00
Adam Simpkins
6cec539d3f fix dirstate.normallookup() behavior
Summary:
Fix `dirstate.normallookup()` to behave like `dirstate.normal()`, since our
`dirstate.status()` function does not handle "unsure" files that need to be
checked on disk.

This was previously causing problems after running `hg commit -i`.

We ideally should probably have some additional changes here to avoid making an
filesystem `lstat()` call even in the `dirstate.normal()`, but that can wait
until later.

Reviewed By: bolinfest

Differential Revision: D6746814

fbshipit-source-id: c1623415fc3a923b0b7878be651db3ad8fbb2aff
2018-01-19 15:16:09 -08:00
Adam Simpkins
319b991379 rename stdout_charset to encoding for hg() and git() functions
Summary:
Rename the `stdout_charset` parameter to `encoding` now that we also use this
for encoding the stdin contents.

Reviewed By: wez

Differential Revision: D6757300

fbshipit-source-id: f79fc760e0f9fdcd6af559fcdad34976aaf16412
2018-01-18 20:24:08 -08:00
Adam Simpkins
6ed5a601cc improve passing stdin to hg in integration tests
Summary:
Update the `hg split` tests to pass input to the split command using
the `input` argument to `subprocess.run()` rather than running the command
through the shell and using a shell "here document".

Reviewed By: bolinfest, chadaustin

Differential Revision: D6746815

fbshipit-source-id: b607c554a249f702758dcad2ac4d863d1c1c7bad
2018-01-18 20:24:08 -08:00
Sergey Zhupanov
0b4fea5374 change eden clone to check out master commit in both git and hg.
Summary:
Changed `eden clone` to check out master commit in both git and hg.
Previously, it checked out the current commit for the repo.

Reviewed By: simpkins

Differential Revision: D6663754

fbshipit-source-id: 92b185ccca5d082dc2bde9c8b191c82a2a4f06b4
2018-01-13 14:26:32 -08:00
Philip Jameson
8604b8f5b0 Migrate TARGETS files from @/ to //
Summary:
This is a codemod to change from using @/ to // in basic cases.
- TARGETS files with lines starting with @/ (but excluding @/third-party:
- autodeps lines in source and TARGETS files ( (dep|manual)=@/ ), excluding @/third-party
- Targets in string macros

The only thing left of the old format should be @/third-party:foo:bar

drop-conflicts

Reviewed By: ttsugriy

Differential Revision: D6605465

fbshipit-source-id: ae50de2e1edb3f97c0b839d4021f38d77b7ab64c
2017-12-20 16:57:41 -08:00
Michael Bolin
33dec91a98 Fix if that was supposed to be elif.
Summary:
Encountered a funny situation where running `hg clone src dest` from inside an
Eden mount where `src` was a directory that contained a non-Eden Hg repo would
fail with a stacktrace that ended with:

```
File "/usr/local/fb-mercurial/eden/hgext3rd/eden/__init__.py", line 195, in merge_update
  conflicts = repo.dirstate.eden_client.checkout(
AttributeError: 'dirstate' object has no attribute 'eden_client'
```

This was very confusing because we had this check at the top of the function:

```
if not util.safehasattr(repo.dirstate, 'eden_client'):
    why_not_eden = 'This is not an eden repository.'
```

So it seemed that we already verified that `repo.dirstate.eden_client` must be a
valid attribute. However, we followed this check with a new set of checks, the
final one being:

```
else:
    why_not_eden = None
```

This one had the unintended effect of resetting the value of `why_not_eden` that
we set in the first `if`. Changing the `if` to an `elif` introduces the proper
decision tree.

Reviewed By: simpkins

Differential Revision: D6608867

fbshipit-source-id: 320e69925737135d84f9d6a46a7fb43437cc37e0
2017-12-19 21:50:09 -08:00
Michael Bolin
c0f3976d57 Add integration test to verify rebasing a stack with commits works as expected.
Summary:
This attempts to repro an internal bug report, though note the integration test
passes without any changes to Eden. It is possible that the person who reported
the bug was stuck on an old version of the Eden daemon while using a newer
version of the Mercurial extension, which could account for the unexpected
behavior.

Reviewed By: simpkins

Differential Revision: D6536375

fbshipit-source-id: 1bc4c50ee5f616502dc06f8ed0167817c566e179
2017-12-13 17:35:51 -08:00
Adam Simpkins
42d5654314 make sure hg update --clean clears merge conflict state
Summary:
Make sure `hg update --clean` clears the merge state data.  In non-clean
updates this is performed in `mercurial.merge.applyupdates()`.  However, we
never call `applyupdates()` on clean updates in eden.

Reviewed By: bolinfest

Differential Revision: D6456720

fbshipit-source-id: b40d02ca0fb677bcde82822a8eafd5fcf926dae6
2017-12-01 16:35:57 -08:00
Michael Bolin
5e2afa735f Change how the UNTRACKED_ADDED conflict and merges are handled.
Summary:
Previously, we used the Mercurial code `g` when faced with an `UNTRACKED_ADDED`
file conflict, but that was allowing merges to silently succeed that should not
have. This revision changes our logic to use the code `m` for merge, which
unearthed that we were not honoring the user's `update.check` setting properly.

Because we use `update.check=noconflict` internally at Facebook, we changed the
Eden integration tests to default to verifying Hg running with this setting. To
support it properly, we had to port this code from `update.py` in Mercurial to
our own `_determine_actions_for_conflicts()` function:

```
if updatecheck == 'noconflict':
    for f, (m, args, msg) in actionbyfile.iteritems():
        if m not in ('g', 'k', 'e', 'r', 'pr'):
            msg = _("conflicting changes")
            hint = _("commit or update --clean to discard changes")
            raise error.Abort(msg, hint=hint)
```

However, this introduced an interesting issue where the `checkOutRevision()`
Thrift call from Hg would update the `SNAPSHOT` file on the server, but
`.hg/dirstate` would not get updated with the new parents until the update
completed on the client. With the new call to `raise error.Abort` on the client,
we could get in a state where the `SNAPSHOT` file had the hash of the commit
assuming the update succeeded, but `.hg/dirstate` reflected the reality where it
failed.

To that end, we changed `checkOutRevision()` to take a new parameter,
`checkoutMode`, which can take on one of three values: `NORMAL`, `DRY_RUN`, and
`FORCE`. Now if the user tries to do an ordinary `hg update` with
`update.check=noconflict`, we first do a `DRY_RUN` and examine the potential
conflicts. Only if the conflicts should not block the update do we proceed with
a call to `checkOutRevision()` in `NORMAL` mode.

To make this work, we had to make a number of changes to `CheckoutAction`,
`CheckoutContext`, `EdenMount`, and `TreeInode` to keep track of the
`checkoutMode` and ensure that no changes are made to the working copy when a
`DRY_RUN` is in effect.

One minor issue (for which there is a `TODO`) is that a `DRY_RUN` will not
report any `DIRECTORY_NOT_EMPTY` conflicts that may exist. As `TreeInode` is
implemented today, it is a bit messy to report this type of conflict without
modifying the working copy along the way.

Finally, any `UNTRACKED_ADDED` conflict should cause an update to
abort to match the behavior in stock Mercurial if the user has the following
config setting:

```
[commands]
update.check = noconflict
```

Though the original name for this setting was:

```
[experimental]
updatecheck = noconflict
```

Although I am on Mercurial 4.4.1, the `update.check` setting does not seem to
take effect when I run the integration tests, but the `updatecheck` setting
does, so for now, I set both in `hg_extension_test_base.py` with a `TODO` to
remove `updatecheck` once I can get `update.check` to do its job.

Reviewed By: simpkins

Differential Revision: D6366007

fbshipit-source-id: bb3ecb1270e77d59d7d9e7baa36ada61971bbc49
2017-11-29 21:50:34 -08:00
Adam Simpkins
c9c60762b0 fix flatmanifest import fallback
Summary:
I accidentally broke the flatmanifest fallback code in D6333613 by changing the
exception type thrown for errors received from hg_import_helper.py but not
updating the catch statement in HgImporter::importTreeImpl().

This updates importTreeImpl() to catch the new HgImportPyError type correctly.
I have dropped the check on the error message entirely, since the mercurial
python code can throw a variety of errors that all mean this tree data isn't
available.

Reviewed By: bolinfest

Differential Revision: D6434359

fbshipit-source-id: c62d3c1667681712293873de2b9bf6d9220da767
2017-11-29 14:36:39 -08:00
Adam Simpkins
a3aa8d11e7 add python type information to more integration test code
Summary: Annotate more integration test functions with type information.

Reviewed By: bolinfest

Differential Revision: D6434358

fbshipit-source-id: b88351eebee58561465752378c6771b7b1f9554e
2017-11-29 14:36:39 -08:00
Michael Bolin
c729a17a11 New condition for test_update_with_untracked_file_that_is_tracked_in_destination.
Summary:
We now run two versions of this test: one where the file that exists in the
destination commit is untracked before the update and one in which it is added
before the update.

Reviewed By: simpkins

Differential Revision: D6334002

fbshipit-source-id: ef6bffa27bc18171b5e21dc284c7a21aa6e35da4
2017-11-20 15:56:35 -08:00
Michael Bolin
ade30224d9 Add assert_dirstate_empty to test_update_clean_removes_added_and_removed_statuses.
Summary: Addresses outstanding TODO now that D6322052 has landed.

Reviewed By: simpkins

Differential Revision: D6368884

fbshipit-source-id: 497c42466e05af0f1690bc6401b1d271de691e58
2017-11-20 14:07:41 -08:00
Michael Bolin
a9d8e55931 Verify that hg update --clean clears the dirstate.
Summary:
Users often run `hg update --clean .` to get out of a bad state, but this was
not clearing the "added" or "removed" state in the dirstate as it should in
Eden.

Reviewed By: wez

Differential Revision: D6331858

fbshipit-source-id: 616f187930587a1af40a1f151e3a424d50dd8da3
2017-11-14 22:26:08 -08:00
Adam Simpkins
60c300b8f3 override dirstate.rebuild()
Summary:
Override dirstate.rebuild() so that it drops all file changes without trying to
mark every file in the commit manifest as normal.  We don't want to track
normal files in the eden dirstate.

Reviewed By: bolinfest

Differential Revision: D6322227

fbshipit-source-id: d81ade1cdafb5fa03c642239b0cff91308c7fc35
2017-11-14 20:07:15 -08:00
Adam Simpkins
645d271d1c implement hg debugdirstate
Summary:
Previously calling `hg debugdirstate` inside an eden repository crashed, since
it would try to iterate over the dirstate and we do not allow this.

This re-implements the `debugdirstate' command inside eden repositories to
print out the data stored in eden's dirstate file.

Reviewed By: wez

Differential Revision: D6322052

fbshipit-source-id: 92f230438a545fe83134b0d67545ebb89877d2f0
2017-11-14 20:07:15 -08:00
Adam Simpkins
a5991f1e4c add type annotations to some more integration test functions
Summary:
Add type annotations necessary to make `mypy --strict` pass cleanly on
`status_deadlock_test.py`

Reviewed By: bolinfest

Differential Revision: D6268257

fbshipit-source-id: b35db4b171c8a3631b5d7a1f1298fb9e68670ff1
2017-11-09 19:05:52 -08:00
Michael Bolin
be4ff47192 New actions that resulted from applyupdates() in merge.py were not getting applied.
Summary:
In the course of verifying a fix for `hg update --merge` in D6270272, I
discovered a new bug in our merge logic in the Python code. As expained in the
test plan, there was a case where a file was listed as "untracked" instead of
"added" after a merge with `--tool :local`.

I traced through what happens in stock Mercurial. After the call to
`applyupdates()` in `update()` in `merge.py`, there is this code:

```
stats = applyupdates(repo, actions, wc, p2, overwrite, labels=labels)
wc.flushall()

if not partial:
    with repo.dirstate.parentchange():
        repo.setparents(fp1, fp2)
        recordupdates(repo, actions, branchmerge)
        # update completed, clear state
        util.unlink(repo.vfs.join('updatestate'))

        if not branchmerge:
            repo.dirstate.setbranch(p2.branch())
```

It turns out that `applyupdates()` can have the side-effect of adding new
entries to the `actions` dict. In this case, we have a `'cd'` action for which
an `'am'` action is generated. Our `merge_update()` function in
`eden/hg/eden/__init__.py` did not have the `recordupdates()` call that the
stock implementation of Mercurial does, so the `'am'` (for "add/merge") was not
getting applied.

It seems likely that introducing this `recordupdates()` call may fix other
subtle bugs in Eden's Mercurial extension for which we do not yet have
integration tests.

Reviewed By: wez

Differential Revision: D6279971

fbshipit-source-id: 901c1bc563a7a3910dde18cf2f0d8b8ff9cd6fbe
2017-11-09 16:29:56 -08:00
Michael Bolin
c6f59d25b8 Fix a crash that could occur when doing hg update .^ --merge.
Summary:
The underlying issue is that we were reporting a `MODIFIED_REMOVED`
conflict as a `MODIFIED_MODIFIED` conflict. This put us in a state where
Mercurial expected to find a file in the new manifest, but failed because the
file was not present in that revision, so no such file could be found.

Somewhat surprisingly, the appropriate handler for a `MODIFIED_REMOVED`
conflict already existed in our Mercurial extension, but there was no logic on
the server that would generate a `MODIFIED_REMOVED` conflict previous to
this change.

Like D6204916, this was an issue I ran into when trying to create a repro case
for the issue that was fixed in D6199215.

Reviewed By: wez

Differential Revision: D6270272

fbshipit-source-id: 6604eea00b0794cd44b01d2ba6b9ea10db32d556
2017-11-09 16:29:55 -08:00
Michael Bolin
a377c7c817 Make it so hg adding an ignored file actually adds it.
Summary:
If a file is explicitly added via `hg add`, then it should be considered added
even if it matches a pattern in `.gitignore`. Further, if it is deleted without
running `hg forget`, it should be considered missing rather than ignored.

To make this work, I had to update `eden_dirstate.walk()`, which already
had a special case when used as part of `hg add`. The new logic ensures
that files that are specified explicitly are still considered even if they are
matched via `.gitignore`.

I also had to address a TODO in `EdenThriftClient.py` related to the
handling of ignored files that was introduced as part of the major
`eden_dirstate` changes in D6179950. It was expected that it would
be easier to handle ignored files properly after D6179950 landed.

Reviewed By: wez

Differential Revision: D6242223

fbshipit-source-id: cf1cfe97a8d2ec57bce1d524074c43978a78e4ef
2017-11-07 19:50:11 -08:00
Adam Simpkins
c3ebc91fdc update hg integration test inheritance to allow type checking
Summary:
Python 3 type checking currently complains about most of our integration
testing since the tests use an `hg_test` decorator to inherit from the base
test class.  This prevents the type checker from being able to figure out this
inheritance.

This updates all of the test cases to explicitly derive from the test case base
class, rather than using the decorator to do so.  I also renamed the base test
case class to `EdenHgTestCase` to be slightly more succinct and to follow the
common pattern of calling `unittest.TestCase` subclasses `FooTestCase`

Reviewed By: bolinfest

Differential Revision: D6268258

fbshipit-source-id: 09eef2f8217932a6516f78d17dddcd35c83b73da
2017-11-07 19:04:20 -08:00
Adam Simpkins
017f636ad3 add an integration test to exercise importing during "hg status"
Summary:
This adds an integration test that exercises a deadlock we could encounter in
the past.  An "hg status" operation could trigger many trees and files to be
imported.  Unfortunately the file import code currently blocks waiting for file
import futures to complete.  This could result in a state where all threads in
the pool were waiting for a file import to complete, and the file import was
waiting for a free thread to complete.

Reviewed By: bolinfest

Differential Revision: D6216871

fbshipit-source-id: e1795a543a71fccbed035febb159e126e27d1950
2017-11-07 19:04:20 -08:00
Michael Bolin
4c24e5bd9f hg prev should leave untracked files alone if no conflicts, even if in a directory.
Summary:
This fixes an issue where the `DIRECTORY_NOT_EMPTY` conflict type reported by
the server was not handled by the client. Somewhat ironically, the fix appears
to be to explicitly "do nothing," though the important part of this revision is
the new integration test.

As this is only one test, I'm not convinced this covers all possible corner
cases, but it's certainly better than blowing up, which is what we did before.

Reviewed By: wez

Differential Revision: D6264069

fbshipit-source-id: a7c45a43776a903a4d6b6cdfb0ce75db9549c380
2017-11-07 19:04:20 -08:00
Michael Bolin
a43eb5afaf Add proper support for the UNTRACKED_ADDED conflict type.
Summary:
If you have an untracked file and you `hg update` to a commit that has
that file in the tracked state, then the contents of the untracked version
should be ignored, as they are replaced with the contents of the file in the
commit you are updating to. The untracked version should be backed up
as specified by `ui.origbackuppath`.

Previously, our code in `eden/hg/eden/__init__.py` mapped this to a merge action
named `c`, but we did not include that in our set of `actions`, so we were
getting a `KeyError` if you exercised this code path.

I discovered this while trying to reproduce the issue that I fixed in D6199215.

Reviewed By: simpkins

Differential Revision: D6204916

fbshipit-source-id: b70153428291bda9a8853a37c0955ad7cb3bd89d
2017-11-07 17:50:52 -08:00
Michael Bolin
5d738193e5 Store Hg dirstate data in Hg instead of Eden.
Summary:
This is a major change to how we manage the dirstate in Eden's Hg extension.

Previously, the dirstate information was stored under `$EDEN_CONFIG_DIR`,
which is Eden's private storage. Any time the Mercurial extension wanted to
read or write the dirstate, it had to make a Thrift request to Eden to do so on
its behalf. The upside is that Eden could answer dirstate-related questions
independently of the Python code.

This was sufficiently different than how Mercurial's default dirstate worked
that our subclass, `eden_dirstate`, had to override quite a bit of behavior.
Failing to manage the `.hg/dirstate` file in a way similar to the way Mercurial
does has exposed some "unofficial contracts" that Mercurial has. For example,
tools like Nuclide rely on changes to the `.hg/dirstate` file as a heuristic to
determine when to invalidate its internal caches for Mercurial data.

Today, Mercurial has a well-factored `dirstatemap` abstraction that is primarily
responsible for the transactions with the dirstate's data. With this split, we can
focus on putting most of our customizations in our `eden_dirstate_map` subclass
while our `eden_dirstate` class has to override fewer methods. Because the
data is managed through the `.hg/dirstate` file, transaction logic in Mercurial that
relies on renaming/copying that file will work out-of-the-box. This change
also reduces the number of Thrift calls the Mercurial extension has to make
for operations like `hg status` or `hg add`.

In this revision, we introduce our own binary format for the `.hg/dirstate` file.
The logic to read and write this file is in `eden/py/dirstate.py`. After the first
40 bytes, which are used for the parent hashes, the next four bytes are
reserved for a version number for the file format so we can manage file format
changes going forward.

Admittedly one downside of this change is that it is a breaking change.
Ideally, users should commit all of their local changes in their existing mounts,
shutdown Eden, delete the old mounts, restart Eden, and re-clone.

In the end, this change deletes a number of Mercurial-specific code and Thrift
APIs from Eden. This is a better separation of concerns that makes Eden more
SCM-agnostic. For example, this change removes `Dirstate.cpp` and
`DirstatePersistance.cpp`, replacing them with the much simpler and more
general `Differ.cpp`. The Mercurial-specific logic from `Dirstate.cpp` that turned
a diff into an `hg status` now lives in the Mercurial extension in
`EdenThriftClient.getStatus()`, which is much more appropriate.

Note that this reverts the changes that were recently introduced in D6116105:
we now need to intercept `localrepo.localrepository.dirstate` once again.

Reviewed By: simpkins

Differential Revision: D6179950

fbshipit-source-id: 5b78904909b669c9cc606e2fe1fd118ef6eaab95
2017-11-06 19:56:49 -08:00
Michael Bolin
95fd684e2e Both ENOTDIR and ENOENT should be ignored when stat'ing a possibly missing file.
Summary:
There is logic in `eden_dirstate.walk()` that looks to see if any of the files
that are reported as "removed" by `hg status` are still on disk, and if so,
should be considered for a walk. Because the files are likely removed, we were
catching `ENOENT` for a failed `os.stat()`, but we also needed to be catching
`ENOTDIR`. This turned out to be the reason `hg add` was failing in a specific
case, for which we already had an integration test, but it wasn't passing until
now.

Reviewed By: simpkins

Differential Revision: D6207233

fbshipit-source-id: 44e5252bb0130ca279160f0a64286053fa5509d5
2017-11-01 20:49:44 -07:00
Michael Bolin
0ec034fde4 Add an integration test for removing a directory and replacing it with a file.
Summary:
I ran into this issue while manually testing Eden.

Currently, this integration test fails, so it is tagged with `unittest.skip`.
There are substantial changes to our distate logic coming in D6179950, so I
will attempt to make the test pass as part of that revision.

Reviewed By: simpkins

Differential Revision: D6199789

fbshipit-source-id: cd7ce48b72bf0b54e13547b23823f4d496fa5b0b
2017-10-31 14:02:57 -07:00
Michael Bolin
ed155d84cc Add p and pr to the list of potential merge actions when dealing with conflicts.
Summary:
Upstream, some new merge actions were added:

* `p` https://phab.mercurial-scm.org/D776
* `pr` https://phab.mercurial-scm.org/D777

We must include entries for these in the list of `actions` that we build up in
`eden/hg/eden/__init__.py` because the `actions` dict gets passed through to
Mercurial's own `applyupdates()` function in `merge.py` that contains this line:

```
for f, args, msg in actions['p']:
```

Therefore, without an entry for `p` in `actions` here, we get a `KeyError`.

Reviewed By: markbt

Differential Revision: D6199215

fbshipit-source-id: a7408e5ef84a659f37e7771a7c15f6a4b14ae0f9
2017-10-31 12:24:16 -07:00
Michael Bolin
28214295bb Add some extra assertions to GraftTest.
Summary:
In practice, if the `hg graft` succeeds in a weird way, `assert_status_empty()`
tells a lot more about what went wrong than the number of commits not matching up.

While here, I also added the following entry to the default `.hgrc` used in integration tests:

```
[ui]
origbackuppath=.hg/origbackups
```

I needed this for the change to `graft_test.py`. As we were already setting this option in
the `histedit_command.py` utility as a one-off and this is the default value of this setting
for our internal Mercurial use at Facebook, it seemed best to make it the default for all
of our integration tests. As such, I removed the one-off setting in `histedit_command.py`.

Reviewed By: simpkins

Differential Revision: D6180342

fbshipit-source-id: 6f0487624a1824459403126997ea52d1a7921feb
2017-10-30 21:38:14 -07:00
Adam Simpkins
74c1027bba flush pending transaction data in eden_dirstate_map.setparents()
Summary:
Previously we flushed the pending transaction data in
eden_dirstate.setparents().  However, some dirstate code paths (particularly
dirstate.rebuild()) can directly call eden_dirstate_map.setparents().

We need to make sure the transaction data is flushed in this case.

Reviewed By: bolinfest

Differential Revision: D6175410

fbshipit-source-id: 256cb07f57ada02d6c1f118ec5075fb8ac93506c
2017-10-27 14:26:30 -07:00
Michael Bolin
ac5b213e92 Include the dirstate tuples and copymap when backing up the dirstate.
Summary:
Previously, the `savebackup()` and `restorebackup()` methods in `eden_dirstate`
only retained the parent commit hashes. With this change, now the dirstate tuples
and entries in the copymap for the dirstate are also included as part of the saved
state.

Failing to restore all of the state caused issues when doing things like aborting
an `hg split`, as observed by one of our users. Although this fix works, we ultimately
plan to move the responsibility for persisting dirstate data out of Eden and into the
Hg extension. Then the data will live in `.hg/dirstate` like it would for the default
dirstate implementation.

Reviewed By: simpkins

Differential Revision: D6145420

fbshipit-source-id: baa077dee73847a47cc171cd980cdd272b3a3a99
2017-10-25 22:36:06 -07:00
Michael Bolin
a286ad53e5 Introduce eden debug hg_dirstate command.
Summary:
Add an `eden debug hg_dirstate` command to dump the contents of the Hg dirstate.
This data is stored in a binary format, so we need a custom command to view it
easily.

Reviewed By: simpkins

Differential Revision: D6139172

fbshipit-source-id: 622c0b7bcaa471a88483c6c4ddef7e0be95a3dfa
2017-10-25 22:36:06 -07:00
Adam Simpkins
2e6ed25612 flush kernel caches properly for empty directories removed by checkout
Summary:
When performing an source control checkout operation, we attempt to remove
directories that are empty after the checkout.  However, this code path was
missing a call to flush the kernel cache for these directories.

As a result, even though eden thought the directory not longer existed, and
would not report it in `readdir()` results, the kernel would return stale
information from its cache when explicitly accessing this path.

Reviewed By: bolinfest

Differential Revision: D6151543

fbshipit-source-id: 6031feb268ff6f980c885efb26c3d43243dec3f4
2017-10-25 16:51:56 -07:00
Michael Bolin
264d6fa9b7 Fix bug where hg histedit --abort does not restore file as "normal".
Summary:
This ports some logic from Mercurial's `dirstate.py` to our `eden_dirstate.py`.
It was known that Eden was missing this behavior (D5686636), but we did not have
an integration test that demonstrated the importance of this logic until now.

Admittedly, the current implementation ports the logic verbatim from
`dirstate.py`, though this will yield quite a number of Thrift calls in the Eden
implementation. We will address this in a subsequent revision.

Reviewed By: simpkins

Differential Revision: D6046273

fbshipit-source-id: f7a27ba6dca36cddac898f19637f29f3bc79a0cb
2017-10-12 19:28:11 -07:00
Michael Bolin
9e8e24d7df Create fix and test for hg merge.
Summary:
Running Mercurial's own integration tests revealed that we had a bug here:
https://www.mercurial-scm.org/repo/hg/file/tip/tests/test-histedit-arguments.t

Somewhat unsurprisingly, it was time to finally address a longstanding `TODO`
in `Dirstate.cpp`. The issue was that, after running `hg merge --tool :local`,
`hg status` was not including a merged file in the list of modified files. Because
the information from `hg status` is used to create a commit context, that meant
that when a commit was made after running `hg merge`, the commit did not
include the merged file in the list of files for the commit, which differs from
Mercurial's behavior.

Most of the implementation of `hg status` on the Eden side is done by
`EdenMount.diff()`. However, in this case, `diff()` does not categorize the
merged file by invoking one of the methods of `InodeDiffCallback` because
as far as `EdenMount` is concerned, the file has not changed because `EdenMount`
is unaware of the `Dirstate`. We already have some existing cases where we have
to do some post-processing on the result of `EdenMount.diff()` using information
in the `Dirstate` (e.g., files that are marked for addition or removal), so the fix was
to add a check for the case when the file is flagged as "needs merging" and
then including it as modified in the `hg status` output, as appropriate.

Reviewed By: wez

Differential Revision: D6005603

fbshipit-source-id: 7d4dd80e1a2e9f4b98243da80989e0e9119a566d
2017-10-09 11:55:34 -07:00
Michael Bolin
ef6f17696e Update RollbackTest to reflect error message change in Mercurial.
Summary:
Note that the original motivation for this test was to verify
`savebackup()` and `restorebackup()` in `eden_dirstate`: D5485950.

As singhsrb recently updated Mercurial to remove a redundant commit when doing
`hg amend` in upstream Mercurial (https://phab.mercurial-scm.org/D636), I
suspect that is responsible for the change in behavior that is necessitates the
change in our test.

We now use a precommit hook failure to trigger the rollback rather than an editor
with a non-zero exit code. As you can see, `transaction abort!\nrollback completed\n`
still appears in the error message, so we are still verifying the behavior of interest.

Differential Revision: D5826751

fbshipit-source-id: bcbf00042c3f26b6e9aa1a980060a0561725a56c
2017-09-19 19:14:43 -07:00
Michael Bolin
1c6403e435 Fix hg grep so it can be run over the entire repo.
Summary:
Although this is not the type of behavior we want to encourage, we should make
it possible. It turns out that this was throwing an exception becuase
`make_glob_list()` was erroneously mapping the pattern to `/**/*` instead of
`**/*` in this case.

Reviewed By: wez

Differential Revision: D5826753

fbshipit-source-id: 659d67c13cdcda39abb7d6893a57ef046804da73
2017-09-13 17:51:19 -07:00
Michael Bolin
250ee8c1eb Fix hg grep so it works when run from a subdirectory.
Summary:
It turns out that we had a small bug with our matcher code that did not account
for pattern normalization. I discovered this while dogfooding Eden and using
`hg grep <pattern> <directory>` from a subdirectory in my working copy. Given
that the fix was to patterns, in general, this likely fixes other `hg` commands
that take a file pattern when used someplace other than the repo root.

Reviewed By: wez

Differential Revision: D5825483

fbshipit-source-id: 0d639cbb2fc678c5459e02e965bf6fc6d7c10959
2017-09-13 12:57:40 -07:00
Michael Bolin
1c695e6dc6 Fixed a bug in how we update the dirstate upon a snapshot change.
Summary:
Previously, we were clearing entries in `hgDirstateTuples` for which:

```
mergeState == NotApplicable
```

but we should have been checking for:

```
mergeState == NotApplicable AND status == Normal
```

The previous logic was causing us to erroneously clear entries in a state like:

```
mergeState == NotApplicable AND status == MarkedForRemoval
```

This bug manifested itself when grafting a change that removed a file.
The file was removed from disk, but Eden did not know that it had been
`MarkedForRemoval`, so it would report the removed file as "missing" in
`hg status`.

Reviewed By: wez

Differential Revision: D5797270

fbshipit-source-id: 29740dfaa8102db868b95e932716773787f317ac
2017-09-08 19:25:34 -07:00
Michael Bolin
595c2684f0 Print out the reason why we take the slow path during a merge update.
Summary:
This should help us audit the source of the slow path when we hit it.

I took a look at `eden/integration/hg/rebase_test.py`, which we know exercises
the slow path. With this change, I manually rebased a short stack of two commits
onto another stack of two commits with the `--debug` flag and saw two instances
of this message:

```
falling back to non-eden update code path: branchmerge is "truthy:" True.
```

so it seems like we should work to update the `branchmerge` case to take the
fast path, when possible.

Reviewed By: simpkins

Differential Revision: D5779633

fbshipit-source-id: a76d72408d6115aa37ae563d3f7165f404fc8332
2017-09-06 21:20:45 -07:00
Michael Bolin
83b3c38095 Fix for hg split in Eden.
Summary:
Before this change, `hg split` crashed complaining that `node` was a
`changectxwrapper` instead of a 20-byte hash when it was sent as `parent1`
of `WorkingDirectoryParents` in `resetParentCommits()`. Now we use `node()` to
get the hash from the `destctx` that we have already extracted via this line
earlier in `merge_update()`:

    destctx = repo[node]

The change to `eden/hg/eden/__init__.py` eliminated the crash, but was
not sufficient on its own to make `hg split` work correctly. There was also a fix
required in `Dirstate.cpp` where the `onSnapshotChanged()` callback was clearing out
entries of both `NotApplicable` and `BothParents` from `hgDirstateTuples`.
It appears that only `NotApplicable` entries should be cleared. (I tried leaving
`NotApplicable` entries in there, but that broke `eden/integration/hg/graft_test.py`.)

I suspected that the logic to clear out `hgDestToSourceCopyMap` in
`Dirstate::onSnapshotChanged` was also wrong, so I deleted it and all of the
integration tests still pass. Admittedly, we are pretty weak in our test coverage
for use cases that write to the `hgDestToSourceCopyMap`. In general, we should
rely on Mercurial to explicitly remove entries from `hgDestToSourceCopyMap`.
We have a Thrift API, `hgClearDirstate()`, that `eden_dirstate` can use to categorically
clear out `hgDirstateTuples` and `hgDestToSourceCopyMap`, if necessary.

Finally, creating a proper integration test for `hg split` required creating a value for
`HGEDITOR` that could write different commit messages for different commits.
To that end, I added a `create_editor_that_writes_commit_messages()` utility as a
method of `HgExtensionTestBase` and updated its `hg()` method to take `hgeditor`
as an optional parameter.

Reviewed By: wez

Differential Revision: D5758236

fbshipit-source-id: 5cb8bf4207d4e802726cd93108fae4a6d48f45ec
2017-09-06 21:20:45 -07:00
Wez Furlong
79ca5bff6e adjust test expectations for fixed typo
Summary: bolinfest fixed the spelling in upstream hg

Reviewed By: simpkins

Differential Revision: D5712622

fbshipit-source-id: 50f98493483a3371bbd26318507b0bb1dcdc1e6d
2017-08-25 21:57:28 -07:00
Michael Bolin
c2a541f84b Fix typo in comment.
Summary: (Note: this ignores all push blocking failures!)

Reviewed By: simpkins

Differential Revision: D5699075

fbshipit-source-id: 98ca3b395dd9895fe3d0a43e7daaa6bcedd90689
2017-08-24 14:23:39 -07:00