Summary:
Some folks might attempt to load this extension despite it
not being useful with eden. The extension knows not to hook in to
the repo when eden is in used, but the various commands to manipulate
the sparse config do not.
This diff adds a little helper function to show a more reasonable
error message than an ugly stack trace when attempting to use them.
Reviewed By: simpkins
Differential Revision: D8328229
fbshipit-source-id: d7cc4c7047b11f8da96e26ab70bd522b52546151
Summary:
Fix the integration tests to set CHG_BIN correctly when run manually rather
than via `buck test`.
Reviewed By: wez, quark-zju
Differential Revision: D8329593
fbshipit-source-id: 79fa3c9e9538e4b30a98ef777515527562e1ac1b
Summary:
This fixes a bug simpkins pointed out in D6891479 - we weren't
updating mtime and ctime on renames.
Reviewed By: simpkins
Differential Revision: D7937303
fbshipit-source-id: 08fd8f4fe5d99d33e9f7629965d6146330c8f35b
Summary:
I kept running into issues trying to get graceful restart and
flush_cache to work together in the hg integration suite, so add a
test to ensure flush_cache succeeds after a graceful restart in the
main integration suite.
Also, to make the test's output easier to follow, add logging when
invalidating inodes.
Reviewed By: simpkins
Differential Revision: D8215961
fbshipit-source-id: 33db4292af3969ae23940c3027ba513ed20c53fb
Summary:
Add a command line option to start edenfs only if there is at least one
Eden checkout already configured.
This makes it easier to automatically start edenfs on system boot only if the
user needs edenfs.
Reviewed By: wez
Differential Revision: D8174819
fbshipit-source-id: bf585b8c5aa719d3b6424567f2e21c1c63850d66
Summary:
This adds a debug command to blow away all RocksDB information that
can be reproduced from Mercurial. We will use it to help an Eden user
recover from a corrupted blob.
Reviewed By: bolinfest
Differential Revision: D8108649
fbshipit-source-id: 056dec19d51b9e430b3c2a249747b26830cfc875
Summary:
The Overlay is the natural home for nextInodeNumber_ now that every
directory allocates inode numbers for its children right away. This
also simplifies serializing nextInodeNumber_ to disk in the following
diff.
Reviewed By: simpkins
Differential Revision: D8192442
fbshipit-source-id: 9b776a73c8d7653002b55985d592b1746e52f878
Summary:
Add a new integration test that modifies a file, commits it, then restores it
to its original contents and creates a new commit with the revert.
This test isn't really exercising any Eden-specific behavior. The only reason
it is is interesting is that this happened to trigger a bug in the underlying
mercurial cdatapack code that caused the test to fail when run in treemanifest
mode. The mercurial cdatapack bug is fixed in D8131020
Reviewed By: quark-zju
Differential Revision: D7826284
fbshipit-source-id: 32da2db04452d5df0527f6be00b0c4c1a56c900b
Summary:
Update find_executables.py to be able to find the HG wrapper and binary using
buck rules. This is necessary in order to be able to run the integration
tests manually rather than via `buck test`. Without this the tests try to use
`/bin/hg.real` which tests the wrong version of the mercurial code and the
eden extension.
Reviewed By: bolinfest
Differential Revision: D8200168
fbshipit-source-id: d2a68c765c7143bbe613f32eef1268ed567790a7
Summary: Mostly empty lines removed and added. A few bugfixes on excessive line splitting.
Reviewed By: cooperlees
Differential Revision: D8198776
fbshipit-source-id: 4361faf4a2b9347d57fb6e1342c494575f2beb67
Summary:
Relax the restriction on changing uid/gid on inodes. We'll see what
cans of worms this opens I guess. (Landing this is low priority, but
might be important for making some of the existing tooling in fbsource
and www work.)
Reviewed By: simpkins
Differential Revision: D7768655
fbshipit-source-id: 95fe02fe7ddc001335dbdb34e16a989a85820240
Summary:
Add a `remove` command to the Eden CLI. This behaves like
`eden unmount --destroy`, but calling this "remove" is hopefully a more
intuitive UI. If stdin is a TTY this command also prompts the user for
confirmation before removing the checkout.
I plan to deprecate the `eden unmount --destroy` command in a subsequent
diff.
Reviewed By: wez
Differential Revision: D8086823
fbshipit-source-id: 562cf0f998eea416b80589b188eee255a10b9699
Summary: Now that permissions on directories work, verify umask works as intended.
Reviewed By: simpkins
Differential Revision: D7783743
fbshipit-source-id: 635221cd3255cc20e9ffa26b6838922c4a4110f3
Summary:
This moves most inode metadata management into InodeBase and
persists permission bits (and eventually uid/gid) across Eden runs.
Reviewed By: simpkins
Differential Revision: D7035163
fbshipit-source-id: 50145449b56aad1662d53156e6e4960c5f7b6166
Summary: Store tree and file timestamps in the InodeTable so they persist across runs.
Reviewed By: simpkins
Differential Revision: D6891479
fbshipit-source-id: 1c9e6266375aceeaf293a81e73cf7f5334dbc32d
Summary:
This is stacked on top of Black 18.5b0.
allow-large-files
Reviewed By: carljm
Differential Revision: D8061834
fbshipit-source-id: 92e3645e159b60d77cf7e0bec64a8262ca4e88c2
Summary:
This updates the Eden mercurial extension to no longer invoke the Eden
`resetParentCommits()` thrift call when `setparents()` is called on the
dirstate map. Instead we now defer the call to `resetParentCommits()` until
`write()` is called to write the dirstate data to disk.
Informing edenfs of the parent change as soon as `setparents()` was called was
problematic, as this made edenfs reflect the change before the transaction was
committed. Some mercurial commands, notably `hg status` also call
`setparents()` on the dirstate but never write this back to disk at all. This
is problematic since `hg status` calls `setparents()` without holding any
mercurial locks. As a result it may call `setparents()` with the "wrong"
parent if another mercurial process is running and is in the middle of a
transaction.
Reviewed By: bolinfest, chadaustin
Differential Revision: D7980375
fbshipit-source-id: 4f5e4391fd291d4ea5fc93bb9d49ed0380fc1721
Summary:
While running the secfs filesystem validation tests against Eden, I
discovered a test that caused the eden process to abort. I bisected
and found that D7451330 regressed renaming a directory onto an empty
one. This fixes that case.
Reviewed By: simpkins
Differential Revision: D7945727
fbshipit-source-id: 592ede1b391528c02cd12b2b6ebbf3733fe8f503
Summary:
We need to introduce a new `includeDotfiles` option to `glob()`. [As we have
done for all of our Thrift API, to date], rather than define `glob()` so that it
takes a single struct, we specified the parameters individually, so we can no
longer add new params to `glob()`.
In particular, we need to support `includeDotfiles` because we often configure
Buck to use Watchman to implement `glob()` in `BUCK` files, and when Watchman is
used in Eden, it leverages Eden's Thrift API to implement `glob()`. Because
Buck's `glob()` has an `include_dotfiles` option, we must be able to honor it
and pass it all the way through to Eden's `glob()` implementation.
Rather than name the new API `glob2()`, I'm electing to go with `globFiles()`.
(Perhaps once we eliminate all known users of `glob()` in the wild, which
requires turning over the current version of Watchman we have deployed, we can
redefine `glob()` in `eden.thrift` to be the same as `globFiles()` and then
update everyone to use `glob()` again so it has the more intuitive name.)
Reviewed By: wez
Differential Revision: D7748870
fbshipit-source-id: 92438f9c41e4fbdbd6cdccca5fce0e41cc3e9b07
Summary:
When comparing two source control blob hashes, identical hashes can be assumed
to mean that the file contents are equal. However, differing hashes does not
necessarily mean that the file contents differ. In particular, mercurial
hashes history metadata in addition to the file contents when computing the
blob hash.
This updates Eden to always compare the file contents when the source control
blob hashes differ, rather than assuming that the file contents are different.
Reviewed By: wez
Differential Revision: D7825900
fbshipit-source-id: e611124a66cdd5c44589f20d1d4665a603286530
Summary:
While testing the fbclone --eden changes I found that
we were failing to pick up the bind mounts. The root cause of
this is that the default head rev for hg repos is `.`, so if
the source repo is bare then we won't find an `.arcconfig` file
and thus won't be able to set up the bind mounts.
Reviewed By: simpkins
Differential Revision: D7782489
fbshipit-source-id: f41d3a7daf39ecd0946707cb2c3211d70c36eea2
Summary:
Disallow any kind of mutation operation inside of the .eden directory. We had some
code in place to prevent some of this already, but errors (including EPERM) weren't
passed out from unlink and rename out to FUSE.
Reviewed By: simpkins
Differential Revision: D7781691
fbshipit-source-id: aaecf13779eca75d6ee8765fc8bb3727ce9341de
Summary:
By introducing `_assert_glob()`, it eliminates a bit of the boilerplate in the
individual test cases. I think it also makes things easier to read because now
the glob patterns appear before the result of the glob.
Though most importantly, this will help with a subsequent change where I am
going to change the contract of the Thrift API, as now it can be done within
`_assert_glob()`.
Reviewed By: wez
Differential Revision: D7748871
fbshipit-source-id: 9609cde104979e892f0858e7a0c7e53e976ff8e2
Summary:
In the spirit of TDD, I am writing the tests first and annotating them with
`unittest.skip()` so the build still succeeds.
Reviewed By: chadaustin
Differential Revision: D7741507
fbshipit-source-id: 4ede0b933c75f9be4016e399936ff2e47eb4e538
Summary:
I think this makes it easier to reason about the coverage of the space of
inputs to `glob()`.
Reviewed By: chadaustin
Differential Revision: D7741509
fbshipit-source-id: 5882d859df95279189512716004263dd5320ff3f
Summary:
I would like to to test more inputs for `glob()` with different characteristics.
I think this would be more logically organized when divided across a number of
test methods in a single `GlobTest` class.
This revision does a straight move of the eixsting `test_glob()` method without
introducing any new test cases.
Reviewed By: chadaustin
Differential Revision: D7741506
fbshipit-source-id: 141341d74265f3949ed7523f40a56f98d95ee13e
Summary:
I found this while testing the fbclone script; clone was
not performing realpath on the destination path, so we ended up
creating and registering a mount under a non-canonical name.
That doesn't sound so bad except that unmount always canonicalizes
the path to be unmounted and this made it impossible to unmount
such a repo.
Reviewed By: chadaustin
Differential Revision: D7766366
fbshipit-source-id: cbfd484a1481d5082969fc97eaf211c18c0d30be
Summary:
Try treating the argument as a repository path first, rather than a well-known
repository name. If we can find a mercurial or git repository at this path,
look for a .arcconfig file in the repository, and try to identify the
repository project type from this file. Use this project name to load our
default bind-mount and hooks configuration for the repository.
If the argument is not a valid repository path, fall back to treating it as a
well-known project name in our repository configs.
This also updates `eden clone` to print a few more diagnostic messages as it
works. (For instance, printing when it is starting the edenfs daemon, and
reporting the commit ID that was checked out.)
Reviewed By: wez
Differential Revision: D7739917
fbshipit-source-id: ac927b9e93039e4d1b8afa80466c2eee3a8829e9
Summary:
Update the Eden mercurial extension to read the `.eden/root` symlink to
determine what Eden thinks the mount path is. This might be different from
what directory mercurial thinks it is in if a parent directory of the Eden
mount has been bind-mounted to an alternate location.
Maybe in the future we should update thrift clients to pass in the client ID
(currently readable via `.eden/client`) rather than the mount path. That would
make it less likely for clients to accidentally forget to read `.eden/root` and
pass in the wrong mount path.
Reviewed By: wez
Differential Revision: D7705655
fbshipit-source-id: 7bd1e8013b99a52ff06dd45f63d6669b66bdf577
Summary: I forgot to remove this comment when I fixed the test in D7565503.
Reviewed By: chadaustin
Differential Revision: D7716106
fbshipit-source-id: 58acd71ac96560f2cc454b3a7da75bb61e6e71c7
Summary:
Rename the rule for the Eden CLI script from `cli` to `eden`. This way the
generated par file will be named `eden.par` rather than `cli.par`. This seems
like it will be slightly less confusing to new developers starting to work on
Eden--currently it doesn't seem particularly intuitive that `cli.par` gets
installed as `eden`.
Reviewed By: bolinfest
Differential Revision: D7658023
fbshipit-source-id: 7afc5e0b703d02751b509efe42f8b8be1f56bbc1
Summary:
These were making some noise in the Nuclide diagnostics pane.
In the course of making these changes, I ended up reformatting
some files, as well. Perhaps we should flag flake8 and autoformat
violations more aggressively in CI now?
Reviewed By: chadaustin
Differential Revision: D7658030
fbshipit-source-id: b52a29b1cc242967f868dcc8ee46ec1bb9bdfbee
Summary:
Fix a couple issues when when performing an `hg update` with updatecheck set to
`noconflict` and when conflicts are detected:
- Report the files with conflicts. The conflicts may be non-obvious to the
user if the conflicts are in ignored files that are not tracked in the
current commit but are present in the destination commit.
- Do not invoke the preupdate hook or create the .hg/updatestate file until
after we have performed the conflict check. Otherwise the repository will be
marked as in the middle of an unfinished update even though we never
attempted the update itself.
Reviewed By: quark-zju
Differential Revision: D7581782
fbshipit-source-id: e290ebd6a27f228a805bea40aabb3f3cc633cc7d
Summary:
Update the assert_status() function in the integration tests so that it also
checks for unfinished update/rebase/graft/etc operations. We unfortunately
have to manually check for the presence of these files ourselves, since
`hg status` provides no mechanism to report this data when `HGPLAIN` is set.
Reviewed By: quark-zju
Differential Revision: D7581781
fbshipit-source-id: 230234e5b8ce28cd3569cdacced686fed2a9dd32
Summary:
This rewrites the `walk()` and `status()` methods in the `eden_dirstate` class.
The changes to the `status()` function should not have any major behavior
changes: this primarily moves logic out of `EdenThriftClient.getStatus()` and
into `dirstate.status()`
The changes to the `walk()` function do fix a number of bugs. We now implement
walk by combining Eden's `getScmStatus()` results with the dirstate's
non-normal files and the current parent commit's manifest. All of the glob
matching is done purely in python now. This fixes some cases where Eden's glob
handling behavior did not support some glob patterns allowed by mercurial. The
Eden glob() call also always returned ignored files and hidden files (such as
everything inside the `.hg` directory). I believe this behavior is desired by
watchman, but it caused problems for the `walk()` code.
Reviewed By: chadaustin
Differential Revision: D7565503
fbshipit-source-id: deb91b4772501e8fbdba56e5c099a72d55fb61ee
Summary:
Update `eden_dirstate.status()` to invoke the `match.bad()` and
`match.explicitdir()` appropriately when any of the explicitly specified match
paths either don't exist or refer to directories.
Reviewed By: chadaustin
Differential Revision: D7556850
fbshipit-source-id: 2e8d88ff0e3c09cea2f3ef5b18c9a214231815ab
Summary:
Implement `dirstate.matches()` in Eden by calling `matches()` on the parent
commit's context, and then updating that result with the list of non-normal
files from Eden's dirstate.
Reviewed By: chadaustin
Differential Revision: D7556847
fbshipit-source-id: 34e78c0c125cd1b184a7773a149edafdca34fee1
Summary:
Add integration tests to check the behavior of the `hg journal` command in an
Eden checkout.
Reviewed By: wez
Differential Revision: D7512339
fbshipit-source-id: 9a83a1752835bc6ded9f0c3b2cb29ec24a6db631
Summary:
When running the integration tests, set HGRCPATH to point to the configs found
at `scm/hg/fb/staticfiles/etc/mercurial`
This does affect mercurial's behavior in some cases, and therefore required
changes to some of our tests:
- `hg diff` output now uses an output style similar to `git diff`
- merge conflict markers include some extra commit metadata
Reviewed By: wez
Differential Revision: D7512321
fbshipit-source-id: 6141605147797ea8b13fdb5a06f67beaeff6afe7
Summary:
This slightly refactors the way that EdenTestCase and EdenRepoTest initialize
repositories. This removes the `create_repo()` method from EdenTestCase and
replaces it with separate `create_hg_repo()` and `create_git_repo()` methods.
The `EdenRepoTest` subclasses now provide alternate `create_repo()`
implementations instead of `get_repo_class()`.
This cleans up the code a bit, since `create_repo()` no longer takes different
arguments based on what type of repository is being created. This also will
make it easier in upcoming diffs to further customize the logic that occurs in
`create_hg_repo()`.
Reviewed By: chadaustin
Differential Revision: D7512320
fbshipit-source-id: d268b0ac0ffb33f3dfd34f2dd1917d57033c81aa
Summary:
Once a mount point has been unmounted we no longer need to care about
outstanding FUSE reference counts--we can treat them as if they are all zero.
This updates EdenMount to tell the InodeMap when the mount point is unloaded,
and changes InodeMap::unloadInode() to make use of this information when
deciding if it needs to remember the inode information.
Previously InodeMap would save information for inodes with outstanding FUSE
reference counts. Writing all of this state to the overlay could take a
non-trivial amount of time.
Reviewed By: chadaustin
Differential Revision: D7555998
fbshipit-source-id: 0896f867ce850ab3e61c262776d536de003685ff
Summary:
Update the GitRepository and HgRepository classes in the integration test code
to use the find_executable module to find the location of git and hg.
Reviewed By: ryanmce
Differential Revision: D7512037
fbshipit-source-id: 139072c3493f70abd0dc92acab295d707bee8ad6
Summary:
Refactor the find_executables module to only look up executables when they are
needed, instead of performing all look-ups immediately even if the test in
question may not need all of the binaries. Also add a _find_exe() helper
function to eliminate some code duplication.
Reviewed By: ryanmce
Differential Revision: D7512038
fbshipit-source-id: fdfb8ec70b3f6292603826b3fb22c01dbd1f0d72
Summary:
Add an integration test that nonmaterialized files have the same inode
numbers after restarting the Eden mount.
Reviewed By: simpkins
Differential Revision: D7328311
fbshipit-source-id: 989abbed38626839e0badffeac2d86dc46c8dd72
Summary:
We already had type annotations on most of the `hg` integration tests. This
adds them for the top-level (non-source-control-specific) tests.
typeseverywhere
Reviewed By: wez
Differential Revision: D7459281
fbshipit-source-id: 41266b232ded510d6b63dd3e62c272a0cd6a0e1a
Summary:
Implement the tear-down actions using addCleanup() rather than through a
tearDown() function. This simplifies the logic, and also makes it easier to
write the code that can be type-checked more easily. (We no longer need to
initialize some variables to `None` just to help `tearDown()` identify if they
have been set up yet or not.)
Reviewed By: wez
Differential Revision: D7459278
fbshipit-source-id: de32ebfcb5474a9583f2a51cc9277102189cb77e
Summary:
Update the eden_repo_test decorator so that it no longer automatically adds
`EdenRepoTestBase` as a parent class. Individual test classes still specify
`EdenRepoTest` as their parent now.
This enables `mypy` to correctly figure out that the individual test classes
derive from `unittest.TestCase`.
This basically does the same thing as D6268258 for the top-level integration
tests.
Reviewed By: wez
Differential Revision: D7459280
fbshipit-source-id: 5d18bd241dad77d55541ac3fa1d169496ffe7003
Summary:
This integration tests appears to be frequently hanging on sandcastle, and
leaving behind unkillable zombie processes. I'm disabling it for now until we
can figure out why it hangs and fix it and/or make sandcastle smart enough to
be able to successfully tear down the FUSE mounts.
These stuck processes are likely stuck waiting on a kernel lock that can only
be released by using `/sys/fs/fuse/connections/DEVICE_ID/abort`
Reviewed By: wez
Differential Revision: D7440282
fbshipit-source-id: 70034fd65267eff18bf5a052cde8d11e0a66b46b
Summary:
Fix FuseChannel::processSession() to always process all FUSE requests that it
reads. Previously it checked to see if it should stop immediately after
reading FUSE request. It was possible for the old process to successfully read
a FUSE request, see that it was supposed to stop, and then exit this worker
thread without ever processing this FUSE request. This would cause the client
that sent the request to hang indefinitely, since no response would ever be
sent.
Reviewed By: wez
Differential Revision: D7436867
fbshipit-source-id: c58c2f6c49102fa6b66ac83fc1639595a5277ce0
Summary:
Perform cleanup with an `addCleanup()` function rather than in `tearDown()`.
This is slightly simpler, but also avoids throwing an exception in `tearDown()`
if `setUp()` failed before it created the thrift client.
Reviewed By: wez
Differential Revision: D7436636
fbshipit-source-id: fe525c1fac86bb866c560cca0ad940ea87d96f07
Summary:
Update the `eden clone` command to accept extra command line flags telling it
where to find the edenfs executable, as well as additional arguments to pass
to edenfs.
This enables the `test_clone_should_start_daemon()` integration test to pass in
flags to tell it how to start edenfs correctly.
Reviewed By: chadaustin
Differential Revision: D7433367
fbshipit-source-id: 0eddd93d1332e113fb85fa4b8fc87ba51d7eab2c
Summary:
Update test_version() so that it can succeed even if the RPM is not installed
on the local system.
Reviewed By: wez
Differential Revision: D7433366
fbshipit-source-id: 5502cca02f9b481cc136beb665048525bae72807
Summary:
Update FuseChannel to send all invalidation requests in a separate thread.
This eliminates a deadlock that could previously occur during checkout
operations. The invalidation requests would block until they could acquire the
kernel's inode lock on the inode in question. However, the inode lock may
already be held by another thread attempting to perform an unlink() or rename()
call. These FUSE unlink or rename operations would be blocked waiting on
Eden's mount point rename lock, which was acquired by the checkout operation.
Checkout operations now let the invalidations complete asynchronously, but we
wait for all invalidation operations to complete before indicating to our
caller that the checkout has succeeded.
Reviewed By: chadaustin, wez
Differential Revision: D7404971
fbshipit-source-id: 6fa20c00d054e210eb0258d247d083010557f210
Summary:
The status_deadlock_test code has a small helper function used to create
directory trees in the test. This moves that function into a helper module so
we can re-use it in other tests in the future.
Reviewed By: chadaustin
Differential Revision: D7407492
fbshipit-source-id: 257e5a2ce7543bb6cd218b412d165f0fac852970
Summary:
Decouple inode number assignment from materialization status.
The idea is that we will always assign entries an inode number and
track whether an entry is materialized otherwise. This is necessary
to give consistent inode values across remounts.
Reviewed By: simpkins
Differential Revision: D7052470
fbshipit-source-id: 80d3f2a2938463198a3132182537e6223c79d509
Summary:
Add a new integration test that performs a graceful restart after invoking the
getScmStatusBetweenRevisions() thrift call.
Prior to D7341609 this would cause edenfs to crash on shutdown with the error
"!!BUG!! After InodeMap::shutdown() finished, 2 inodes still loaded; they must
all (except the root) have been unloaded for this to succeed!"
Before D7341609 the `EdenMount::diffRevisions()` created a new temporary inode
tree solely to perform a diff. This resulted in multiple root inodes that all
pointed to the same EdenMount, but the EdenMount didn't know about any of these
alternate root inodes. These temporary inode trees never got destroyed,
causing this error on shutdown.
Reviewed By: chadaustin, wez
Differential Revision: D7333005
fbshipit-source-id: 8406d2e2ceb00264050b0aceec583baae2da69ec
Summary:
Update the getScmStatusBetweenRevisions() thrift call to use the new
diffCommits() function that diffs source control Tree objects without creating
TreeInode objects to perform the diff.
This addresses two bugs:
- Each call to EdenMount::diffRevisions() constructed a new root inode
associated with the mount, and this would never get destroyed. It was not
destroyed at the end of the diffRevisions() call since inodes are normally
not destroyed immediately when they are unreferenced. It was not destroyed
during EdenMount::shutdown() since EdenMount didn't have any references to
these additional root inode structures and their children.
- EdenMount::diffRevisions() incorrectly swapped ADDED and REMOVED statuses in
the result.
Reviewed By: wez
Differential Revision: D7341609
fbshipit-source-id: 16e755a0ff685f51c977c3b27d6af96908f33494
Summary:
Update Eden's thrift service handler code to accept BinaryHash arguments either
as 20-byte binary values or as 40-byte hexadecimal values.
This will make it easier to transition APIs like getScmStatusBetweenRevisions()
to use 20-byte binary hash arguments without breaking existing clients.
Reviewed By: wez
Differential Revision: D7341607
fbshipit-source-id: 3e952211900d3ec4b9c2073cf3afd55ae7e253ea
Summary:
Add an integration test for the getScmStatusBetweenRevisions() thrift call.
This call apparently gets the ADDED and REMOVED states backwards. For now the
test checks for the current (incorrect) behavior.
This also fixes the thrift definition for this function to stop using the
BinaryHash typedef. Unlike most of our other thrift functions this method
appears to require the arguments as 40-byte hexadecimal strings.
Reviewed By: wez
Differential Revision: D7341606
fbshipit-source-id: 73cbd0ecf4445da6b1f0ef9cf6d9dce47e6fb593
Summary:
This makes gitrepo.commit() return the new commit ID as a hexadecimal string,
just like hgrepo.commit() currently does.
Reviewed By: wez
Differential Revision: D7341605
fbshipit-source-id: 83ebddb8c23d5e4650432bea6f8dcb8d18c0ff38
Summary:
Update the hgrepo and gitrepo helper classes with a new `remove_file()` method,
and also improve the git code to include the process stderr in the exception
message if the command fails.
Reviewed By: wez
Differential Revision: D7341610
fbshipit-source-id: 28cca89520923d92bba4833a4dcfab6d21357cfb
Summary:
Update repobase.py, hgrepo.py, and gitrepo.py to include python type hints for
all functions and to pass most of mypy's strict type checking.
There are still a couple minor errors around the fact that hgrepo.hg() and
gitrepo.git() return an `Optional[str]` rather than a plain `str`. Many callers
know that they cannot return None unless the stdout argument is None, but mypy
can't figure this out. In the long run maybe we should split these into two
separate methods, one that always returns `str` and one that always returns
`None`.
Reviewed By: wez
Differential Revision: D7341608
fbshipit-source-id: c62da578fb32edb9272363fadabbdc11b1d5d2c2
Summary: Fix a simple TODO in EdenServiceHandler.cpp
Reviewed By: chadaustin
Differential Revision: D7329635
fbshipit-source-id: b42b2cc13be3ad5b18a629ed15f6c51cea52fbda
Summary:
If you run `eden daemon` on a machine where sudo needs input, sudo's
output would get redirected to edenfs.log and eden daemon would appear
to hang. Worse, if you pressed ctrl-C, sudo would remain in the
background and continue to swallow keypresses from the tty, appearing
to somewhat break your shell until it gave up. The fix is to stop
redirecting stdout and stderr from Python and instead have edenfs
redirect itself into the log if given.
Reviewed By: simpkins
Differential Revision: D7223399
fbshipit-source-id: bae17b150b6594a3dd87bb88741c8dcefd1c5213
Summary:
This integration test verifies that observed timestamps should persist
across runs. D6891479 makes it pass.
Reviewed By: wez
Differential Revision: D6930208
fbshipit-source-id: b8c95bce00933b9ae0de101a0bd8b6abfbfa1177
Summary:
The getpath_unloaded_inode tests failed on my machine quite
regularly. The two possible races here I can imagine are racing the
system clock and FUSE not having released its refcount on an inode by
the time unload is called.
Reviewed By: wez
Differential Revision: D7118883
fbshipit-source-id: c3708f14a860f5ad04ddec988fc67a683b7dcfde
Summary:
Previously we returned an ENOENT error in response to a FUSE lookup() call for
a name that does not exist. However, this does not allow FUSE to cache the
result, so we will continue to receive lookup() calls for this path in the
future.
This changes EdenDispatcher to return a successful response with an inode
number of 0 instead. This tells the kernel that the name does not exist, but
allows the kernel to cache this negative lookup result (for as long as
specified in the entry_valid field in the response).
Reviewed By: wez
Differential Revision: D7076811
fbshipit-source-id: a2b9977e58d6b6eecb584699b9d93b5ad29ad5ad
Summary:
Add a test that exercises `hg pull`. This confirms that eden can see new
commits created on the server after Eden and its hg_import_helper processes
have started. This test gets run in flatmanifest, hybrid treemanifest, and
treeonly mode.
This currently performs pulls using a local peer repository rather than over
SSH. This does exercise a different code path in mercurial than what typically
occurs in production. In the future we should perhaps also add a test that
uses a fake SSH helper program to exercise mercurial's sshpeer code paths as
well.
Reviewed By: chadaustin
Differential Revision: D6993788
fbshipit-source-id: 40628c0b3faac0dc8622b605a29b084979b8c089
Summary:
Whenever we tell Eden to change the working directory
parents, we need to make sure the appropriate objects are written to
disk. This fixes hg fold in Eden.
Reviewed By: simpkins
Differential Revision: D7045299
fbshipit-source-id: cbd51be59cf943a843b77c2abe66a84b745bce22
Summary:
Update the integration tests to avoid specifying an explicit path to the eden
extension. This way they use the version that we now package into hg.par
during the build.
This avoids issues with hg not being able to find and load native .so libraries
from the eden extension. Mercurial is able to find these libraries correctly
when they are packaged into hg.par (since the par start-up script sets
LD_LIBRARY_PATH to point to the par unpack directory). When using eden from an
external directory mercurial was not able to find these libraries.
Reviewed By: chadaustin
Differential Revision: D7047245
fbshipit-source-id: d56bffa953c178949c866efec507298a1f40da8b
Summary:
This removes the TARGETS files from the eden github repository. The
open source buck build has been failing for several months, since buck
removed support for the thrift_library() rule.
I will potentially take a stab at adding CMake build support for Eden
at some point in the future.
Reviewed By: chadaustin
Differential Revision: D6893233
fbshipit-source-id: e6023094a807cf481ac49998c6f21b213be6c288
Summary:
Add a command line flag to control whether or not Eden should ever try falling
back to import tree data using flatmanifest if an error occurs trying to import
it directly via treemanifest, in repositories that support treemanifest.
This is particularly useful for tests, where we usually do not want to fall
back to flatmanifest import if an error occurs during treemanifest import. The
fallback can otherwise mask real issues that should trigger test failures.
This is probably also a good thing to have in general. Supporting
flatmanifest+treemanifest data in a single Eden repository has some unfortunate
problems today: we compute hashes differently for flatmanifest trees vs
treemanifest trees. As a result, we can end up with identical trees that have
different hashes. This can result in unfortunate performance consequences in
some cases where Eden assumes it must scan a directory for differences if the
hashes are different.
I have left flatmanifest import enabled by default for now, but we may want to
disable it by default in the future. I would be more inclined to disable it by
default if we did added a thrift method to explicitly re-enable it (or to
import a single commit using flatmanifest), so that users could work around
this setting if necessary without having to fully restart edenfs.
Reviewed By: wez
Differential Revision: D6993791
fbshipit-source-id: 6e091a426cf1e7c973df5a641d2f8a1101011346
Summary:
Update the integration tests to build `hg_import_helper` into a python archive
that includes the current mercurial sources from the local repository. This
way hg_import_helper will use the local mercurial code rather than whatever
mercurial modules are installed on the system. This will help ensure that we
detect any breakages caused by changes in the mercurial source when the
mercurial changes are made rather than when they are deployed.
Reviewed By: wez
Differential Revision: D6993790
fbshipit-source-id: f3ad404583cadcf07156bac1ce6bc869bd1160e1
Summary:
Update the `hg_test` decorator to accept additional parameters specifying the
list of test modes. e.g., `hg_test('Treemanifest')` asks to only run the test
in the Treemanifest configuration. With no arguments tests are still run with
both the Flatmanifest and Treemanifest configurations by default.
This also enables the TreeOnly configuration mode, which appears to work now.
(It was previously disabled since `hg init` would fail in treeonly
repositories.)
This new changes allows tests to explicitly opt-in to running in `TreeOnly`
mode.
Reviewed By: wez
Differential Revision: D6993789
fbshipit-source-id: 9ee51318d0f661038fe29f246b2b14eebbb1c3d9
Summary:
Update the integration tests so that they do not delete the temporary test
directories if the environment variable `EDEN_TEST_NO_CLEANUP` is set.
This makes it easier to manually examine the repository state after a test
fails.
Reviewed By: chadaustin
Differential Revision: D6986217
fbshipit-source-id: 727321c2c3da4d19d9edf8ed20b2aca3449779de
Summary:
Update the logic for how the Eden integration tests find the hg binary:
- Use the contents of the EDEN_HG_BINARY environment variable if set. When
running tests via `buck test` buck will pass the hg.par output location in
this variable.
- If EDEN_HG_BINARY is not set, use libfb.py.pathutils to find the location of
the //scm/hg:hg rule output. This makes sure the integration tests still
prefer this par path even when run manually without EDEN_HG_BINARY set. This
is convenient when running individual tests not through buck.
If for some reason the hg python_binary() output cannot be found then we still
search through $PATH for hg.real or hg as usual. For internal fbsource builds
we generally shouldn't hit this fallback case, though.
Reviewed By: wez, quark-zju
Differential Revision: D6986221
fbshipit-source-id: 982cb99112405a674dbc45df4ada73a990536489
Summary:
We would immediately satisfy the health check and
tell the user that the system is healthy and show the pid of
the prior incarnation rather than the one for the instance
that we just launched.
This diff refactors the health checking code so that we can
share the implementation between the cli and the integration
tests; the integration tests already had code to do the right
thing for this.
Reviewed By: simpkins
Differential Revision: D6944989
fbshipit-source-id: 7c0f02c875b1b81f8f1b7521add67928200b27ed
Summary:
chadaustin is going to think about how to test this
in unit tests intestead :-p
Reviewed By: chadaustin
Differential Revision: D6951788
fbshipit-source-id: 137f285f3a1f080ce43392a621c73640ce3a9bf7
Summary:
This makes the default `memory` for speed and minimal
flakiness, but allows a test to select a different engine where
appropriate (eg: restart and remount tests).
Reviewed By: chadaustin
Differential Revision: D6944207
fbshipit-source-id: 1fb11387beda02d059a796dad5a42d56ddcf6e88
Summary:
Adds a SQLite storage implementation and makes it the
default engine for integration tests; this requires fewer resources
to run and the integration tests thus run faster and more reliably.
In the future we may add a configuration option to remember the
storage engine that was used as it is currently not "safe" to switch
from one to the other because the hgproxyhash data cannot be
recreated without re-importing a revision.
Reviewed By: simpkins
Differential Revision: D6919456
fbshipit-source-id: 3afbfafb190cca0e3c797cd9b7cd051768575a8c
Summary:
To me, restart implied shutting down the daemon and restarting it
again. Perhaps instead of `eden daemon --takeover` we should have
`eden restart`. But if people typed `eden restart` I imagine they're
trying to debug a problem, so that's probably not the right verb.
Reviewed By: wez
Differential Revision: D6929166
fbshipit-source-id: d568a1940d67f755e4c3656098c58fc81e0a3156
Summary: This should make it less flaky
Reviewed By: simpkins
Differential Revision: D6908454
fbshipit-source-id: 07622b469ec635972290c3a4e5d1ed4b05ddd51a
Summary:
This was causing flakiness in the form of premature timeouts
generating failure status. Looking at the docs for hypothesis, this
option is deprecated and slated to be replaced with an infinite timeout
in the future:
http://hypothesis.readthedocs.io/en/latest/settings.html#hypothesis.settings.timeout
Those docs suggest using `hypothesis.unlimited` instead, but I couldn't import
that symbol (maybe our version is too old?) so I'm just using 0 which means
that no timeout will be applied.
Reviewed By: simpkins
Differential Revision: D6908200
fbshipit-source-id: 1143d4745fc84d9c14b29ff5a2906a4e776d14fe
Summary:
A little bit of plumbing to enable the watchman<->eden integration tests on sandcastle.
I noticed that we were still referencing the lame thrift par that I removed a little while back,
so let's clean that up.
Reviewed By: simpkins
Differential Revision: D6863221
fbshipit-source-id: 026f3d316d9ac52fe359cedc4e3c88394ec5c7c4
Summary:
This is a fairly minimal attempt at asking sandcastle to
run our tests as root. This is inspired by D6608746 but is for
the moment pretty coarse; I just want to see how successful this
is before finishing this off nicely, which means splitting our
contbuild config into two pieces along similar lines as D6608746;
one piece would run the tests as usual without root privileges
whilst the other would run just the integration tests with root
privs.
Reviewed By: chadaustin
Differential Revision: D6612669
fbshipit-source-id: 9a730b02424e939dd3509310b6265d50d1102e77
Summary: Rather than EPERM we get EACCESS when running as root
Reviewed By: chadaustin
Differential Revision: D6853303
fbshipit-source-id: c6743c8d036a24255d2d31c560034c972a0253c7
Summary:
in our sandcastle environment, eden is not currently
installed and the tests that look at the version are throwing
an error when the rpm query fails.
Don't do that!
Reviewed By: simpkins
Differential Revision: D6853305
fbshipit-source-id: d17cb1fb0253ef5ae5000695b6ec5bcda4a6a448
Summary:
This is the spiritual successor to D3302706 which originally
wanted to solve this by adding a python extension. That would prove
to be too painful for the opensource build so it was shelved.
We now need to be able to run our tests in an environment that doesn't
have the `attr` rpm installed so this is a good time to fix this
in a more portable way.
This diff adds a little wrapper around the functions that we already
have for consuming extended attribute information and augments them
with another to list attributes.
The utility emits output in json format and is intended to be fed
directly into the helper functions we have in `fs.py`.
Reviewed By: chadaustin
Differential Revision: D6851182
fbshipit-source-id: 3d1d1a351f2e01405645d45658d1c8bc61a659a4
Summary:
isSameAs calls getSha1 which was failing on symlinks. The
original concern was that asking for the SHA-1 of a symlink is
ambiguous: do you want the hash of the symlink or the target? But we
already check for whether you are requesting the SHA-1 of a symlink in
EdenServiceHandler, so it's redundant and incorrect to check in
FileInode too.
Reviewed By: simpkins
Differential Revision: D6847489
fbshipit-source-id: 13966da06bcde75c5c568e09fef14e735de47cfb
Summary:
This adds support for the following flags for `hg status` in the Rust wrapper:
* `--all`
* `--modified`
* `--added`
* `--removed`
* `--deleted`
* `--clean`
* `--unknown`
* `--ignored`
* `--no-status`
* `--copies`
There is now a `create_print_config()` function that takes the parsed arguments
and distills them into a `PrintConfig`, which describes the net configuration
as a result of all the flags. (Note that some flags can interfere with each other,
e.g., `--no-status` forces `--copies` to be disabled.)
While here, I cleaned up a bunch of duplication in the code that prints the
status to stdout. Much of the duplication in the old implementation has been
encapsulated in a closure within the new `print_status()` function.
This also made it easier to implement `--copies` for all of the different status
types.
Incidentally, this helps us specify `listIgnored` correctly when calling the
`getScmStatus()` Thrift method as we now keep track of whether the user
specified `--ignored`. This is particularly important for the Eden integration
test suite because its `assert_status()` helper method calls
`hg status --print0 -mardui` by default, so `buck test eden/...` makes much
greater use of the Rust wrapper than it did before.
Reviewed By: wez
Differential Revision: D6770916
fbshipit-source-id: fea1a159addca6d16caa1b7019dc11dc2dcb3e1b
Summary: Add type annotations to the test functions in this file.
Reviewed By: chadaustin
Differential Revision: D6789863
fbshipit-source-id: d298a388b5ea03fcf25047937e747051c8fa682b
Summary:
Test running `hg commit` with a path argument. This currently fails if the
path refers to a directory rather than a file.
Reviewed By: chadaustin
Differential Revision: D6789862
fbshipit-source-id: 054ee26ea0fdb843cc12d5e8ac3c3d34b9308017
Summary: removed check for target dir of eden clone in edenclient
Reviewed By: simpkins
Differential Revision: D6770615
fbshipit-source-id: 4f480c2a494673179d92c0571082b9be895cf51b
Summary:
Update the `hg update --merge` to test marking the change resolved and
committing the result.
Reviewed By: wez
Differential Revision: D6757299
fbshipit-source-id: c147e6ff8b63e364f96d2ce8a2094dd809d8da7d
Summary:
Fix `dirstate.normallookup()` to behave like `dirstate.normal()`, since our
`dirstate.status()` function does not handle "unsure" files that need to be
checked on disk.
This was previously causing problems after running `hg commit -i`.
We ideally should probably have some additional changes here to avoid making an
filesystem `lstat()` call even in the `dirstate.normal()`, but that can wait
until later.
Reviewed By: bolinfest
Differential Revision: D6746814
fbshipit-source-id: c1623415fc3a923b0b7878be651db3ad8fbb2aff
Summary:
The storage format for dirstate data in Eden changed substantially in D6179950.
Although `status.rs` was modified as part of that change, it was only updated to
make it so the rust wrapper still compiled, but it did not maintain the
correctness of the overall function.
This revision is a port of the logic in `eden/py/dirstate.py` and
`eden/hg/eden/EdenThriftClient.py` so that the Python code in the Eden/Hg
extension behaves the same as that of the Rust wrapper.
The current implementation of `hg status` in Rust does not relativize paths to `$PWD`, so
it always behaves as if `--root-relative` was specified. This should be fixed in a
follow-up revision.
Reviewed By: jsgf
Differential Revision: D6717211
fbshipit-source-id: ca900e251d392a77c07cb7305c4d4a62def1c7ab
Summary: Added integration test validating umask is properly set on file creation by touch.
Reviewed By: wez
Differential Revision: D6749557
fbshipit-source-id: adb89ec996148ede95bdb23399745b39504ae8db
Summary:
Rename the `stdout_charset` parameter to `encoding` now that we also use this
for encoding the stdin contents.
Reviewed By: wez
Differential Revision: D6757300
fbshipit-source-id: f79fc760e0f9fdcd6af559fcdad34976aaf16412
Summary:
Update the `hg split` tests to pass input to the split command using
the `input` argument to `subprocess.run()` rather than running the command
through the shell and using a shell "here document".
Reviewed By: bolinfest, chadaustin
Differential Revision: D6746815
fbshipit-source-id: b607c554a249f702758dcad2ac4d863d1c1c7bad
Summary:
Added --version to main.py, including -v and version.
It prints both installed and running version (as per Wez's suggestion), and accounts for the possibilities that eden may not be running at all, or dev version may be running.
Reviewed By: wez
Differential Revision: D6724204
fbshipit-source-id: 5085f53a00a557f759a23fe41fb57189c9ad6a7e
Summary:
Changed `eden clone` to check out master commit in both git and hg.
Previously, it checked out the current commit for the repo.
Reviewed By: simpkins
Differential Revision: D6663754
fbshipit-source-id: 92b185ccca5d082dc2bde9c8b191c82a2a4f06b4
Summary:
This implements a TODO/FATAL that is important for
graceful restarts to be useful in my "acid test" scenario,
which is to perform a graceful restart while buck build is
running.
Reviewed By: simpkins
Differential Revision: D6700189
fbshipit-source-id: dec1b818ebc9e907841bc127ee08c953b59d6487
Summary:
Fix lines longer than 80 characters, and also reduce the number of times the
initial directory listing is repeated throughout this file.
Reviewed By: wez
Differential Revision: D6710372
fbshipit-source-id: bdb02cbebabeff7d7c6c88aebee7ebab1865535b
Summary:
Update the statfs() code to return non-zero values for namelen and frsize.
Returning 0 for namelen was causing problems for programs that checked
`pathconf(path, _PC_NAME_MAX)` and tried to honor this value. For instance,
GNU patch would try to generate files with a 0-length name since we indicated
the maximum name length was 0.
I haven't investigated too closely, but this behavior might have broken only
recently when we stopped using libfuse. libfuse may have been setting this to
255 for us. I didn't see code in libfuse that would do this, but I'm fairly
sure GNU patch was working correctly very recently.
Reviewed By: wez
Differential Revision: D6710370
fbshipit-source-id: fc9a0320cd7c1eb2545219a3ec123c2f0644fb5d
Summary:
I'm so-so on a bit of the implementation here, but it works!
I had to change the `takeoverPromise` from the `pair<fuseDevice, connInfo>`
to a new helper struct because we now have three distinct pieces of data
to pass out of EdenMount to build up the overall TakeoverData.
The key change in this diff is that we have to release all of the file handles
we're maintaining in the `FileHandleMap` prior to shutting down the `InodeMap`,
otherwise the `InodeMap` will never complete (it waits for all inodes to be
unreferenced, and that cannot happen while there are open file handles). I've
made the `FileHandleMap` serialization and clearing contingent on performing a
takeover shutdown because that feels like the safest thing to do wrt. not
losing any pending writes.
Reviewed By: simpkins
Differential Revision: D6672437
fbshipit-source-id: 7b1f0f8e7ff09dbed850c7737383ecdf1e5ff0c7
Summary:
This is the key portion that makes the graceful restart
function. This diff connects almost all of the moving pieces together;
it informs the priv helper about the takeover mount and transfers
the InodeMap information into the new generation of the eden server.
It doesn't yet load the fileHandleMap (will tackle that in a follow up diff)
Reviewed By: simpkins
Differential Revision: D6670903
fbshipit-source-id: 1770d99eb1477440a6c1deed83b0da55b9c1bbe4
Summary:
The unix `patch` command attempts to issue an `FS_IOC_GETFLAGS`
ioctl and blows up when we return `ENOSYS`. The man page for ioctl says:
```
ENOTTY The specified request does not apply to the kind of object that
the file descriptor fd references.
```
so let's return that error code to ioctl.
In addition, the integration test I added for this trips up when it calls
`llistxattr` on the file; turns out we don't need to insert the
`fuse_getxattr_out` when we're returning the attribute list and that it
is only needed when measuring up the required length, so let's move
things around to resolve this and make it clearer.
Reviewed By: chadaustin
Differential Revision: D6685568
fbshipit-source-id: 81963ffe9af30db5634e5e96b7a8aa1485859d65
Summary:
Update the integration tests to understand the new `eden list` output format
changes introduced in D6661303.
Reviewed By: chadaustin
Differential Revision: D6665272
fbshipit-source-id: 13f65a7a66f997e6b8135436811fe3fc396eb38f
Summary:
We're unconditionally emitting a `--` above, so the `--`
down here doesn't get stripped out by the logic in `do_daemon`
in the CLI and that renders it invisible to gflags:
Here's our `eden cli` invocation:
```
['/data/users/wez/eden-fbsource/fbcode/buck-out/gen/eden/cli/cli.par', '--config-dir', '/var/tmp/eden_test.15oico01/homedir/local/.eden', '--etc-eden-dir', '/var/tmp/eden_test.15oico01/etc-ed
en', '--home-dir', '/var/tmp/eden_test.15oico01/homedir', 'daemon', '--daemon-binary', '/data/users/wez/eden-fbsource/fbcode/buck-out/gen/eden/fs/service/edenfs', '--foreground', '--', '--num
_hg_import_threads', '2', '--', '--logging=eden.fs.store.hg=DBG9,eden.strace=DBG7']
```
This is what `eden cli` passes to `edenfs`:
```
edenfs_args ['--num_hg_import_threads', '2', '--', '--logging=eden.fs.store.hg=DBG9,eden.strace=DBG7']
```
Reviewed By: bolinfest
Differential Revision: D6628253
fbshipit-source-id: 2c6806e69baff52d14ca64194f1bf7d916833844
Summary:
This is a codemod to change from using @/ to // in basic cases.
- TARGETS files with lines starting with @/ (but excluding @/third-party:
- autodeps lines in source and TARGETS files ( (dep|manual)=@/ ), excluding @/third-party
- Targets in string macros
The only thing left of the old format should be @/third-party:foo:bar
drop-conflicts
Reviewed By: ttsugriy
Differential Revision: D6605465
fbshipit-source-id: ae50de2e1edb3f97c0b839d4021f38d77b7ab64c
Summary:
Encountered a funny situation where running `hg clone src dest` from inside an
Eden mount where `src` was a directory that contained a non-Eden Hg repo would
fail with a stacktrace that ended with:
```
File "/usr/local/fb-mercurial/eden/hgext3rd/eden/__init__.py", line 195, in merge_update
conflicts = repo.dirstate.eden_client.checkout(
AttributeError: 'dirstate' object has no attribute 'eden_client'
```
This was very confusing because we had this check at the top of the function:
```
if not util.safehasattr(repo.dirstate, 'eden_client'):
why_not_eden = 'This is not an eden repository.'
```
So it seemed that we already verified that `repo.dirstate.eden_client` must be a
valid attribute. However, we followed this check with a new set of checks, the
final one being:
```
else:
why_not_eden = None
```
This one had the unintended effect of resetting the value of `why_not_eden` that
we set in the first `if`. Changing the `if` to an `elif` introduces the proper
decision tree.
Reviewed By: simpkins
Differential Revision: D6608867
fbshipit-source-id: 320e69925737135d84f9d6a46a7fb43437cc37e0
Summary:
My devvm was running out of memory when running `buck test eden/...`.
Because Buck farms out tests across cores and each Eden integration
test launches 8 hg import daemons, there were over a hundred importer
processes live on the system.
Reviewed By: bolinfest, wez
Differential Revision: D6598276
fbshipit-source-id: d26916af79c24aa73abfa4c3ef9be3178657b6e7
Summary:
This attempts to repro an internal bug report, though note the integration test
passes without any changes to Eden. It is possible that the person who reported
the bug was stuck on an old version of the Eden daemon while using a newer
version of the Mercurial extension, which could account for the unexpected
behavior.
Reviewed By: simpkins
Differential Revision: D6536375
fbshipit-source-id: 1bc4c50ee5f616502dc06f8ed0167817c566e179
Summary:
I am seeing occasional timeouts on my smaller devvm when
running `buck test eden/integration/...`.
Reviewed By: bolinfest
Differential Revision: D6541864
fbshipit-source-id: 401deb8b44adae8cc362bbba8b638fe08abb9b1e
Summary:
In D6446057, I added a new entry to the dict returned by
`config.get_client_info()`. I only ran the cli tests while working on D6446057,
but I should have ran all of the tests because there was an integration test
(`InfoTest`) that verified the return value of this method, so it broke due to
my change.
(Note: this ignores all push blocking failures!)
Reviewed By: simpkins
Differential Revision: D6464806
fbshipit-source-id: 1b0ac0853301ba33e5e948353e4c89c0d97c0d83
Summary:
Make sure `hg update --clean` clears the merge state data. In non-clean
updates this is performed in `mercurial.merge.applyupdates()`. However, we
never call `applyupdates()` on clean updates in eden.
Reviewed By: bolinfest
Differential Revision: D6456720
fbshipit-source-id: b40d02ca0fb677bcde82822a8eafd5fcf926dae6
Summary:
Previously, we used the Mercurial code `g` when faced with an `UNTRACKED_ADDED`
file conflict, but that was allowing merges to silently succeed that should not
have. This revision changes our logic to use the code `m` for merge, which
unearthed that we were not honoring the user's `update.check` setting properly.
Because we use `update.check=noconflict` internally at Facebook, we changed the
Eden integration tests to default to verifying Hg running with this setting. To
support it properly, we had to port this code from `update.py` in Mercurial to
our own `_determine_actions_for_conflicts()` function:
```
if updatecheck == 'noconflict':
for f, (m, args, msg) in actionbyfile.iteritems():
if m not in ('g', 'k', 'e', 'r', 'pr'):
msg = _("conflicting changes")
hint = _("commit or update --clean to discard changes")
raise error.Abort(msg, hint=hint)
```
However, this introduced an interesting issue where the `checkOutRevision()`
Thrift call from Hg would update the `SNAPSHOT` file on the server, but
`.hg/dirstate` would not get updated with the new parents until the update
completed on the client. With the new call to `raise error.Abort` on the client,
we could get in a state where the `SNAPSHOT` file had the hash of the commit
assuming the update succeeded, but `.hg/dirstate` reflected the reality where it
failed.
To that end, we changed `checkOutRevision()` to take a new parameter,
`checkoutMode`, which can take on one of three values: `NORMAL`, `DRY_RUN`, and
`FORCE`. Now if the user tries to do an ordinary `hg update` with
`update.check=noconflict`, we first do a `DRY_RUN` and examine the potential
conflicts. Only if the conflicts should not block the update do we proceed with
a call to `checkOutRevision()` in `NORMAL` mode.
To make this work, we had to make a number of changes to `CheckoutAction`,
`CheckoutContext`, `EdenMount`, and `TreeInode` to keep track of the
`checkoutMode` and ensure that no changes are made to the working copy when a
`DRY_RUN` is in effect.
One minor issue (for which there is a `TODO`) is that a `DRY_RUN` will not
report any `DIRECTORY_NOT_EMPTY` conflicts that may exist. As `TreeInode` is
implemented today, it is a bit messy to report this type of conflict without
modifying the working copy along the way.
Finally, any `UNTRACKED_ADDED` conflict should cause an update to
abort to match the behavior in stock Mercurial if the user has the following
config setting:
```
[commands]
update.check = noconflict
```
Though the original name for this setting was:
```
[experimental]
updatecheck = noconflict
```
Although I am on Mercurial 4.4.1, the `update.check` setting does not seem to
take effect when I run the integration tests, but the `updatecheck` setting
does, so for now, I set both in `hg_extension_test_base.py` with a `TODO` to
remove `updatecheck` once I can get `update.check` to do its job.
Reviewed By: simpkins
Differential Revision: D6366007
fbshipit-source-id: bb3ecb1270e77d59d7d9e7baa36ada61971bbc49
Summary:
I accidentally broke the flatmanifest fallback code in D6333613 by changing the
exception type thrown for errors received from hg_import_helper.py but not
updating the catch statement in HgImporter::importTreeImpl().
This updates importTreeImpl() to catch the new HgImportPyError type correctly.
I have dropped the check on the error message entirely, since the mercurial
python code can throw a variety of errors that all mean this tree data isn't
available.
Reviewed By: bolinfest
Differential Revision: D6434359
fbshipit-source-id: c62d3c1667681712293873de2b9bf6d9220da767
Summary: Annotate more integration test functions with type information.
Reviewed By: bolinfest
Differential Revision: D6434358
fbshipit-source-id: b88351eebee58561465752378c6771b7b1f9554e
Summary:
There's a bug in some combination of Eden and FUSE where open(O_TRUNC)
followed by a sequence of writes over an existing file does not flush
the kernel's VFS page cache, which manifests as an mmap larger than
the file's size not zeroing the data beyond the file's size. These
tests attempt capture that use case, but they are fiddly.
Disabling ATOMIC_O_TRUNC seems to resolve the issue.
Reviewed By: wez
Differential Revision: D6430152
fbshipit-source-id: f7626e268e778ebab60c66322e0ce42bce746ae1
Summary:
In onboarding users, we usually tell them to run `eden clone fbsource`,
but that fails because users generally have not run `eden daemon` yet.
The simplest thing is to do it for them when they run `eden clone` when
the daemon is not running.
Reviewed By: wez
Differential Revision: D6357249
fbshipit-source-id: dc112c1efe214485e3c5c8e06522d299a100d3a0
Summary:
We now run two versions of this test: one where the file that exists in the
destination commit is untracked before the update and one in which it is added
before the update.
Reviewed By: simpkins
Differential Revision: D6334002
fbshipit-source-id: ef6bffa27bc18171b5e21dc284c7a21aa6e35da4
Summary: Addresses outstanding TODO now that D6322052 has landed.
Reviewed By: simpkins
Differential Revision: D6368884
fbshipit-source-id: 497c42466e05af0f1690bc6401b1d271de691e58
Summary:
This begins implementing the "client-side" portion of graceful takeover in
edenfs. When the --takeover flag is specified, if edenfs finds that another
edenfs process is already running it will attempt to gracefully take over its
mount points rather than exiting with an error.
This does not yet actually take over the mount points themselves--it still
sends dummy mount information during shutdown, and does not use this data
during startup. However, we do perform takeover of the eden lock file and the
thrift server socket.
Reviewed By: bolinfest
Differential Revision: D6038944
fbshipit-source-id: 42406a0559367cec79af088b4ca84c22de3f3ef3
Summary:
When cloning an existing Eden mount, we should be smart and inherit its
underlying config so that we inherit properties such as its bind mounts.
Reviewed By: wez
Differential Revision: D6322002
fbshipit-source-id: 3f5ba135b12ad7dcecef6676d27495cfbf0ce97b
Summary:
Previously, a user had to define a config for a repo in a file like `~/.edenrc`
in order to create a new Eden mount via `eden clone`. In practice, the
information that is hardcoded in the config can generally be inferred from an
existing repo, so this expands `eden clone` to support both modes of operation.
Note this made it possible to finally unify the `RepoConfig` and `ClientConfig`
types. This revision removes `RepoConfig`, so I dutifully renamed every
local variable named `repo_config` to `client_config`.
Reviewed By: wez
Differential Revision: D6314115
fbshipit-source-id: 9625a5fbe35b30f76b6099180580c64435a4cf72
Summary:
The headline changes of this revision are:
- Changes the format of the config file from INI to TOML
(the `edenrc` file under `~/local/.eden` has been replaced
with `config.toml`). This revision includes logic for automatically
performing the migration when Eden is restarted.
- Inlines data from `/etc/eden/config.d` into the TOML file.
Historically, the `edenrc` file for a client would contain the
name of the "configuration alias" defined in a config file like
`~/.edenrc` or `/etc/eden/config.d/00-defaults`. When Eden
loaded a client, it would have to first read the `edenrc` and
then reconstitute the rest of the client configuration by
looking up the alias in the set of config files that were used to
create the client in the first place.
This changes things so that all of the data that was being
cross-referenced is now inlined in the client's config file.
This makes loading a config considerably simpler at the cost
of no longer being able to change the config for multiple clients
that were cloned from the same configuration alias in one place.
It was questionable whether being able to modify a client from
a foreign config after it was created was a safe thing to do, anyway.
Eliminating the need for a historic link to the configuration alias
will make it easier to support running `eden clone` on an arbitrary
local Hg or Git repo. So long as `eden clone` can extract enough
information from the local repo to create an appropriate config file
for the new Eden client, there is no need for a configuration alias
to exist a priori.
Since we were already changing the data in the config file, this
seemed like an appropriate time to make the switch from INI to
TOML, as this was something we wanted to do, anyway.
In testing, I discovered a discrepancy between how boost's
`boost::property_tree::ptree` and Python's `ConfigParser` handled
the following section heading:
```
[repository ZtmpZsillyZeden-clone.LIkh32]
```
Apparently `hasSection("repository ZtmpZsillyZeden-clone.LIkh32")`
in boost would fail to find this section. Because
[[https://stackoverflow.com/questions/13109506/are-hyphens-allowed-in-section-definitions-in-ini-files | there is no spec for INI]],
it is not that surprising that boost and `ConfigParser` do not 100% agree
on what they accept. Moving to TOML means we have a configuration
language with the following desirable properties:
- It has a formal spec, unlike INI. This is important because there are parsers
in a wide range of programming languages that, in theory, accept a consistent
input language.
- It is reasonable for humans to write, as it supports comments, unlike JSON.
- It supports nested structures, like maps and arrays, without going crazy
on the input language it supports, unlike YAML.
Eden now depends on the following third-party TOML parsers:
* C++ https://github.com/skystrife/cpptoml
* Python https://github.com/uiri/toml
This revision also changes the organization of `~/local/.eden` slightly. For now,
there is still a `config.json` file, but the values are no longer hashes of the realpath
of the mount. Instead, we take the basename of the realpath and use that as the
name of the directory under `~/local/.eden/clients`. If there is a naming collision, we
add the first available integral suffix. Using the basename makes it easier to
navigate the `~/local/.eden/clients` directory.
Although the `edenrc` file under `~/local/.eden/clients` has been switched from INI
to TOML, the other Eden config files (`~/.edenrc` and `/etc/eden/config.d/*`) still use
INI. Migrating those to TOML will be done in a future revision.
Note this revision allowed us to eliminate `facebook::eden::InterpolatedPropertyTree`
as well as a number of uses of boost due to the elimination of
`ClientConfig::loadConfigData()` in the C++ code. Because `ClientConfig`
no longer does interpolation, a bit of `ClientConfigTest` was deleted as part of
this revision because it is no longer relevant.
Reviewed By: wez
Differential Revision: D6310325
fbshipit-source-id: 2548149c064cdf8e78a3b3ce6fe667ff70f94f84
Summary:
This flips the sense of the unmount command; previously
we would default to destroying the mount and associated state,
but this was a bit of a massive sharp edge to our UX.
Now the default is non-destructive and you have to explicitly
pass in `--destroy` to enable destructive mode.
Reviewed By: chadaustin
Differential Revision: D6346013
fbshipit-source-id: ce612e7d8a70540d63217a97f96bc5760f3951af
Summary:
Users often run `hg update --clean .` to get out of a bad state, but this was
not clearing the "added" or "removed" state in the dirstate as it should in
Eden.
Reviewed By: wez
Differential Revision: D6331858
fbshipit-source-id: 616f187930587a1af40a1f151e3a424d50dd8da3
Summary:
Override dirstate.rebuild() so that it drops all file changes without trying to
mark every file in the commit manifest as normal. We don't want to track
normal files in the eden dirstate.
Reviewed By: bolinfest
Differential Revision: D6322227
fbshipit-source-id: d81ade1cdafb5fa03c642239b0cff91308c7fc35
Summary:
Previously calling `hg debugdirstate` inside an eden repository crashed, since
it would try to iterate over the dirstate and we do not allow this.
This re-implements the `debugdirstate' command inside eden repositories to
print out the data stored in eden's dirstate file.
Reviewed By: wez
Differential Revision: D6322052
fbshipit-source-id: 92f230438a545fe83134b0d67545ebb89877d2f0
Summary:
In the course of verifying a fix for `hg update --merge` in D6270272, I
discovered a new bug in our merge logic in the Python code. As expained in the
test plan, there was a case where a file was listed as "untracked" instead of
"added" after a merge with `--tool :local`.
I traced through what happens in stock Mercurial. After the call to
`applyupdates()` in `update()` in `merge.py`, there is this code:
```
stats = applyupdates(repo, actions, wc, p2, overwrite, labels=labels)
wc.flushall()
if not partial:
with repo.dirstate.parentchange():
repo.setparents(fp1, fp2)
recordupdates(repo, actions, branchmerge)
# update completed, clear state
util.unlink(repo.vfs.join('updatestate'))
if not branchmerge:
repo.dirstate.setbranch(p2.branch())
```
It turns out that `applyupdates()` can have the side-effect of adding new
entries to the `actions` dict. In this case, we have a `'cd'` action for which
an `'am'` action is generated. Our `merge_update()` function in
`eden/hg/eden/__init__.py` did not have the `recordupdates()` call that the
stock implementation of Mercurial does, so the `'am'` (for "add/merge") was not
getting applied.
It seems likely that introducing this `recordupdates()` call may fix other
subtle bugs in Eden's Mercurial extension for which we do not yet have
integration tests.
Reviewed By: wez
Differential Revision: D6279971
fbshipit-source-id: 901c1bc563a7a3910dde18cf2f0d8b8ff9cd6fbe
Summary:
The underlying issue is that we were reporting a `MODIFIED_REMOVED`
conflict as a `MODIFIED_MODIFIED` conflict. This put us in a state where
Mercurial expected to find a file in the new manifest, but failed because the
file was not present in that revision, so no such file could be found.
Somewhat surprisingly, the appropriate handler for a `MODIFIED_REMOVED`
conflict already existed in our Mercurial extension, but there was no logic on
the server that would generate a `MODIFIED_REMOVED` conflict previous to
this change.
Like D6204916, this was an issue I ran into when trying to create a repro case
for the issue that was fixed in D6199215.
Reviewed By: wez
Differential Revision: D6270272
fbshipit-source-id: 6604eea00b0794cd44b01d2ba6b9ea10db32d556
Summary:
If a file is explicitly added via `hg add`, then it should be considered added
even if it matches a pattern in `.gitignore`. Further, if it is deleted without
running `hg forget`, it should be considered missing rather than ignored.
To make this work, I had to update `eden_dirstate.walk()`, which already
had a special case when used as part of `hg add`. The new logic ensures
that files that are specified explicitly are still considered even if they are
matched via `.gitignore`.
I also had to address a TODO in `EdenThriftClient.py` related to the
handling of ignored files that was introduced as part of the major
`eden_dirstate` changes in D6179950. It was expected that it would
be easier to handle ignored files properly after D6179950 landed.
Reviewed By: wez
Differential Revision: D6242223
fbshipit-source-id: cf1cfe97a8d2ec57bce1d524074c43978a78e4ef
Summary:
Python 3 type checking currently complains about most of our integration
testing since the tests use an `hg_test` decorator to inherit from the base
test class. This prevents the type checker from being able to figure out this
inheritance.
This updates all of the test cases to explicitly derive from the test case base
class, rather than using the decorator to do so. I also renamed the base test
case class to `EdenHgTestCase` to be slightly more succinct and to follow the
common pattern of calling `unittest.TestCase` subclasses `FooTestCase`
Reviewed By: bolinfest
Differential Revision: D6268258
fbshipit-source-id: 09eef2f8217932a6516f78d17dddcd35c83b73da
Summary:
This adds an integration test that exercises a deadlock we could encounter in
the past. An "hg status" operation could trigger many trees and files to be
imported. Unfortunately the file import code currently blocks waiting for file
import futures to complete. This could result in a state where all threads in
the pool were waiting for a file import to complete, and the file import was
waiting for a free thread to complete.
Reviewed By: bolinfest
Differential Revision: D6216871
fbshipit-source-id: e1795a543a71fccbed035febb159e126e27d1950
Summary:
This fixes an issue where the `DIRECTORY_NOT_EMPTY` conflict type reported by
the server was not handled by the client. Somewhat ironically, the fix appears
to be to explicitly "do nothing," though the important part of this revision is
the new integration test.
As this is only one test, I'm not convinced this covers all possible corner
cases, but it's certainly better than blowing up, which is what we did before.
Reviewed By: wez
Differential Revision: D6264069
fbshipit-source-id: a7c45a43776a903a4d6b6cdfb0ce75db9549c380
Summary:
A few fixes:
* Fix a bug where `date_str` would not get set when `date` was specified.
* Remove `from __future__ import` stuff since this code is Python 3.
* Add type annotations to the `commit()` method.
Reviewed By: simpkins
Differential Revision: D6261874
fbshipit-source-id: 5f942d01c107cd0265c2d6ec6e1f46295bb3ec24
Summary:
If you have an untracked file and you `hg update` to a commit that has
that file in the tracked state, then the contents of the untracked version
should be ignored, as they are replaced with the contents of the file in the
commit you are updating to. The untracked version should be backed up
as specified by `ui.origbackuppath`.
Previously, our code in `eden/hg/eden/__init__.py` mapped this to a merge action
named `c`, but we did not include that in our set of `actions`, so we were
getting a `KeyError` if you exercised this code path.
I discovered this while trying to reproduce the issue that I fixed in D6199215.
Reviewed By: simpkins
Differential Revision: D6204916
fbshipit-source-id: b70153428291bda9a8853a37c0955ad7cb3bd89d
Summary:
Since Eden's integration tests do not run in CI yet, this adds a
test that verifies the Eden CLI can start without Python errors.
Reviewed By: wez
Differential Revision: D6250515
fbshipit-source-id: 907bffaff122c9929a7623d97f665de5b2a6f2d3
Summary:
This is a major change to how we manage the dirstate in Eden's Hg extension.
Previously, the dirstate information was stored under `$EDEN_CONFIG_DIR`,
which is Eden's private storage. Any time the Mercurial extension wanted to
read or write the dirstate, it had to make a Thrift request to Eden to do so on
its behalf. The upside is that Eden could answer dirstate-related questions
independently of the Python code.
This was sufficiently different than how Mercurial's default dirstate worked
that our subclass, `eden_dirstate`, had to override quite a bit of behavior.
Failing to manage the `.hg/dirstate` file in a way similar to the way Mercurial
does has exposed some "unofficial contracts" that Mercurial has. For example,
tools like Nuclide rely on changes to the `.hg/dirstate` file as a heuristic to
determine when to invalidate its internal caches for Mercurial data.
Today, Mercurial has a well-factored `dirstatemap` abstraction that is primarily
responsible for the transactions with the dirstate's data. With this split, we can
focus on putting most of our customizations in our `eden_dirstate_map` subclass
while our `eden_dirstate` class has to override fewer methods. Because the
data is managed through the `.hg/dirstate` file, transaction logic in Mercurial that
relies on renaming/copying that file will work out-of-the-box. This change
also reduces the number of Thrift calls the Mercurial extension has to make
for operations like `hg status` or `hg add`.
In this revision, we introduce our own binary format for the `.hg/dirstate` file.
The logic to read and write this file is in `eden/py/dirstate.py`. After the first
40 bytes, which are used for the parent hashes, the next four bytes are
reserved for a version number for the file format so we can manage file format
changes going forward.
Admittedly one downside of this change is that it is a breaking change.
Ideally, users should commit all of their local changes in their existing mounts,
shutdown Eden, delete the old mounts, restart Eden, and re-clone.
In the end, this change deletes a number of Mercurial-specific code and Thrift
APIs from Eden. This is a better separation of concerns that makes Eden more
SCM-agnostic. For example, this change removes `Dirstate.cpp` and
`DirstatePersistance.cpp`, replacing them with the much simpler and more
general `Differ.cpp`. The Mercurial-specific logic from `Dirstate.cpp` that turned
a diff into an `hg status` now lives in the Mercurial extension in
`EdenThriftClient.getStatus()`, which is much more appropriate.
Note that this reverts the changes that were recently introduced in D6116105:
we now need to intercept `localrepo.localrepository.dirstate` once again.
Reviewed By: simpkins
Differential Revision: D6179950
fbshipit-source-id: 5b78904909b669c9cc606e2fe1fd118ef6eaab95
Summary:
There is logic in `eden_dirstate.walk()` that looks to see if any of the files
that are reported as "removed" by `hg status` are still on disk, and if so,
should be considered for a walk. Because the files are likely removed, we were
catching `ENOENT` for a failed `os.stat()`, but we also needed to be catching
`ENOTDIR`. This turned out to be the reason `hg add` was failing in a specific
case, for which we already had an integration test, but it wasn't passing until
now.
Reviewed By: simpkins
Differential Revision: D6207233
fbshipit-source-id: 44e5252bb0130ca279160f0a64286053fa5509d5
Summary:
Relying on the toString of an Exception in Python seems a little gross,
especially when the `stderr` field is available directly. Cleaned up two
instances of this so it doesn't get copypasta'd further.
Reviewed By: simpkins
Differential Revision: D6195633
fbshipit-source-id: 9ae77796c287a454cb169ebf6de2953909a1e6c3
Summary:
This was an error that an end-user ran into. Previously, we did not fail
gracefully and the user was faced with an intimidating stacktrace.
Reviewed By: simpkins
Differential Revision: D6195529
fbshipit-source-id: bde3c2a3e6f49457a4c6ac5c87103cf52cd227c2
Summary:
I ran into this issue while manually testing Eden.
Currently, this integration test fails, so it is tagged with `unittest.skip`.
There are substantial changes to our distate logic coming in D6179950, so I
will attempt to make the test pass as part of that revision.
Reviewed By: simpkins
Differential Revision: D6199789
fbshipit-source-id: cd7ce48b72bf0b54e13547b23823f4d496fa5b0b
Summary:
Upstream, some new merge actions were added:
* `p` https://phab.mercurial-scm.org/D776
* `pr` https://phab.mercurial-scm.org/D777
We must include entries for these in the list of `actions` that we build up in
`eden/hg/eden/__init__.py` because the `actions` dict gets passed through to
Mercurial's own `applyupdates()` function in `merge.py` that contains this line:
```
for f, args, msg in actions['p']:
```
Therefore, without an entry for `p` in `actions` here, we get a `KeyError`.
Reviewed By: markbt
Differential Revision: D6199215
fbshipit-source-id: a7408e5ef84a659f37e7771a7c15f6a4b14ae0f9
Summary:
In practice, if the `hg graft` succeeds in a weird way, `assert_status_empty()`
tells a lot more about what went wrong than the number of commits not matching up.
While here, I also added the following entry to the default `.hgrc` used in integration tests:
```
[ui]
origbackuppath=.hg/origbackups
```
I needed this for the change to `graft_test.py`. As we were already setting this option in
the `histedit_command.py` utility as a one-off and this is the default value of this setting
for our internal Mercurial use at Facebook, it seemed best to make it the default for all
of our integration tests. As such, I removed the one-off setting in `histedit_command.py`.
Reviewed By: simpkins
Differential Revision: D6180342
fbshipit-source-id: 6f0487624a1824459403126997ea52d1a7921feb
Summary:
Previously we flushed the pending transaction data in
eden_dirstate.setparents(). However, some dirstate code paths (particularly
dirstate.rebuild()) can directly call eden_dirstate_map.setparents().
We need to make sure the transaction data is flushed in this case.
Reviewed By: bolinfest
Differential Revision: D6175410
fbshipit-source-id: 256cb07f57ada02d6c1f118ec5075fb8ac93506c
Summary:
Previous to this change, the user got an inscrutable error message. It turns out
that it is easy to make this mistake, typing `eden clone fbsource/` instead of
`eden clone fbsource` if you accidentally use tab completion.
Reviewed By: simpkins
Differential Revision: D6153889
fbshipit-source-id: 3642fdd207d6abf896d6a12891d5eb68ad984acc
Summary:
Previously, the `savebackup()` and `restorebackup()` methods in `eden_dirstate`
only retained the parent commit hashes. With this change, now the dirstate tuples
and entries in the copymap for the dirstate are also included as part of the saved
state.
Failing to restore all of the state caused issues when doing things like aborting
an `hg split`, as observed by one of our users. Although this fix works, we ultimately
plan to move the responsibility for persisting dirstate data out of Eden and into the
Hg extension. Then the data will live in `.hg/dirstate` like it would for the default
dirstate implementation.
Reviewed By: simpkins
Differential Revision: D6145420
fbshipit-source-id: baa077dee73847a47cc171cd980cdd272b3a3a99
Summary:
Add an `eden debug hg_dirstate` command to dump the contents of the Hg dirstate.
This data is stored in a binary format, so we need a custom command to view it
easily.
Reviewed By: simpkins
Differential Revision: D6139172
fbshipit-source-id: 622c0b7bcaa471a88483c6c4ddef7e0be95a3dfa
Summary:
When performing an source control checkout operation, we attempt to remove
directories that are empty after the checkout. However, this code path was
missing a call to flush the kernel cache for these directories.
As a result, even though eden thought the directory not longer existed, and
would not report it in `readdir()` results, the kernel would return stale
information from its cache when explicitly accessing this path.
Reviewed By: bolinfest
Differential Revision: D6151543
fbshipit-source-id: 6031feb268ff6f980c885efb26c3d43243dec3f4
Summary:
This ports some logic from Mercurial's `dirstate.py` to our `eden_dirstate.py`.
It was known that Eden was missing this behavior (D5686636), but we did not have
an integration test that demonstrated the importance of this logic until now.
Admittedly, the current implementation ports the logic verbatim from
`dirstate.py`, though this will yield quite a number of Thrift calls in the Eden
implementation. We will address this in a subsequent revision.
Reviewed By: simpkins
Differential Revision: D6046273
fbshipit-source-id: f7a27ba6dca36cddac898f19637f29f3bc79a0cb
Summary:
Running Mercurial's own integration tests revealed that we had a bug here:
https://www.mercurial-scm.org/repo/hg/file/tip/tests/test-histedit-arguments.t
Somewhat unsurprisingly, it was time to finally address a longstanding `TODO`
in `Dirstate.cpp`. The issue was that, after running `hg merge --tool :local`,
`hg status` was not including a merged file in the list of modified files. Because
the information from `hg status` is used to create a commit context, that meant
that when a commit was made after running `hg merge`, the commit did not
include the merged file in the list of files for the commit, which differs from
Mercurial's behavior.
Most of the implementation of `hg status` on the Eden side is done by
`EdenMount.diff()`. However, in this case, `diff()` does not categorize the
merged file by invoking one of the methods of `InodeDiffCallback` because
as far as `EdenMount` is concerned, the file has not changed because `EdenMount`
is unaware of the `Dirstate`. We already have some existing cases where we have
to do some post-processing on the result of `EdenMount.diff()` using information
in the `Dirstate` (e.g., files that are marked for addition or removal), so the fix was
to add a check for the case when the file is flagged as "needs merging" and
then including it as modified in the `hg status` output, as appropriate.
Reviewed By: wez
Differential Revision: D6005603
fbshipit-source-id: 7d4dd80e1a2e9f4b98243da80989e0e9119a566d
Summary:
Note that the original motivation for this test was to verify
`savebackup()` and `restorebackup()` in `eden_dirstate`: D5485950.
As singhsrb recently updated Mercurial to remove a redundant commit when doing
`hg amend` in upstream Mercurial (https://phab.mercurial-scm.org/D636), I
suspect that is responsible for the change in behavior that is necessitates the
change in our test.
We now use a precommit hook failure to trigger the rollback rather than an editor
with a non-zero exit code. As you can see, `transaction abort!\nrollback completed\n`
still appears in the error message, so we are still verifying the behavior of interest.
Differential Revision: D5826751
fbshipit-source-id: bcbf00042c3f26b6e9aa1a980060a0561725a56c
Summary:
Although this is not the type of behavior we want to encourage, we should make
it possible. It turns out that this was throwing an exception becuase
`make_glob_list()` was erroneously mapping the pattern to `/**/*` instead of
`**/*` in this case.
Reviewed By: wez
Differential Revision: D5826753
fbshipit-source-id: 659d67c13cdcda39abb7d6893a57ef046804da73
Summary:
It turns out that we had a small bug with our matcher code that did not account
for pattern normalization. I discovered this while dogfooding Eden and using
`hg grep <pattern> <directory>` from a subdirectory in my working copy. Given
that the fix was to patterns, in general, this likely fixes other `hg` commands
that take a file pattern when used someplace other than the repo root.
Reviewed By: wez
Differential Revision: D5825483
fbshipit-source-id: 0d639cbb2fc678c5459e02e965bf6fc6d7c10959
Summary:
Previously, we were clearing entries in `hgDirstateTuples` for which:
```
mergeState == NotApplicable
```
but we should have been checking for:
```
mergeState == NotApplicable AND status == Normal
```
The previous logic was causing us to erroneously clear entries in a state like:
```
mergeState == NotApplicable AND status == MarkedForRemoval
```
This bug manifested itself when grafting a change that removed a file.
The file was removed from disk, but Eden did not know that it had been
`MarkedForRemoval`, so it would report the removed file as "missing" in
`hg status`.
Reviewed By: wez
Differential Revision: D5797270
fbshipit-source-id: 29740dfaa8102db868b95e932716773787f317ac
Summary:
This should help us audit the source of the slow path when we hit it.
I took a look at `eden/integration/hg/rebase_test.py`, which we know exercises
the slow path. With this change, I manually rebased a short stack of two commits
onto another stack of two commits with the `--debug` flag and saw two instances
of this message:
```
falling back to non-eden update code path: branchmerge is "truthy:" True.
```
so it seems like we should work to update the `branchmerge` case to take the
fast path, when possible.
Reviewed By: simpkins
Differential Revision: D5779633
fbshipit-source-id: a76d72408d6115aa37ae563d3f7165f404fc8332
Summary:
Before this change, `hg split` crashed complaining that `node` was a
`changectxwrapper` instead of a 20-byte hash when it was sent as `parent1`
of `WorkingDirectoryParents` in `resetParentCommits()`. Now we use `node()` to
get the hash from the `destctx` that we have already extracted via this line
earlier in `merge_update()`:
destctx = repo[node]
The change to `eden/hg/eden/__init__.py` eliminated the crash, but was
not sufficient on its own to make `hg split` work correctly. There was also a fix
required in `Dirstate.cpp` where the `onSnapshotChanged()` callback was clearing out
entries of both `NotApplicable` and `BothParents` from `hgDirstateTuples`.
It appears that only `NotApplicable` entries should be cleared. (I tried leaving
`NotApplicable` entries in there, but that broke `eden/integration/hg/graft_test.py`.)
I suspected that the logic to clear out `hgDestToSourceCopyMap` in
`Dirstate::onSnapshotChanged` was also wrong, so I deleted it and all of the
integration tests still pass. Admittedly, we are pretty weak in our test coverage
for use cases that write to the `hgDestToSourceCopyMap`. In general, we should
rely on Mercurial to explicitly remove entries from `hgDestToSourceCopyMap`.
We have a Thrift API, `hgClearDirstate()`, that `eden_dirstate` can use to categorically
clear out `hgDirstateTuples` and `hgDestToSourceCopyMap`, if necessary.
Finally, creating a proper integration test for `hg split` required creating a value for
`HGEDITOR` that could write different commit messages for different commits.
To that end, I added a `create_editor_that_writes_commit_messages()` utility as a
method of `HgExtensionTestBase` and updated its `hg()` method to take `hgeditor`
as an optional parameter.
Reviewed By: wez
Differential Revision: D5758236
fbshipit-source-id: 5cb8bf4207d4e802726cd93108fae4a6d48f45ec
Summary:
I had this set and it broke some of the integration tests.
Force it to be unset before running the tests.
Reviewed By: simpkins
Differential Revision: D5712624
fbshipit-source-id: 7d4aef86ef56f5880180b417e356e8a85abf11d7
Summary:
This test was supposed to be a part of D5627411 but it was causing strange behaviour so was brought to a separate diff for further investigation.
After investigating, the test didn't pass because the UnloadedInodeData struct only contained the name of the file, not the path to it. The fix for this was to implement a way to get the relative path of the file even after the inode is unloaded.
Reviewed By: simpkins
Differential Revision: D5646929
fbshipit-source-id: f166398a651e8aea49da7e4474a5ad7fde2eaa4e
Summary:
Fortunately, this passed on the first try: it did not require any bug fixes in
Eden!
(Note: this ignores all push blocking failures!)
Reviewed By: simpkins
Differential Revision: D5698953
fbshipit-source-id: c5ce39725f8d14b5ea93bd3cafeb5e566f92d326
Summary:
Fortunately, this passed on the first try: it did not require any bug fixes in
Eden! Though admittedly, most of the relevant fixes were presumably done in
D5686114.
(Note: this ignores all push blocking failures!)
Reviewed By: simpkins
Differential Revision: D5696055
fbshipit-source-id: 0099db501ae1a5d72528d222dee0176fc1fc4332
Summary:
Update the integration test framework so that we can run the hg integration
tests with several different hg config settings, using different sets of
mercurial extensions.
This adds code to test using flat manifest, treemanifest in hybrid mode, and
treemanifest in tree only mode. However, the treeonly configuration is
disabled at the moment due to some bugs in treeonly behavior preventing it from
being able to create test repositories in treeonly mode.
Reviewed By: bolinfest
Differential Revision: D5685880
fbshipit-source-id: 081ead4e77cd14a7feb03381783395bd5a8fef4f
Summary:
The most recent mercurial release updated the 'successors()' revset to be the
same as 'allsuccessors()', and it always includes the argument itself in the
output now. This updates the revset to exclude the input commit.
Reviewed By: bolinfest
Differential Revision: D5694826
fbshipit-source-id: 3e931a39675262f33a5298701b4559e0d9906490
Summary:
This makes some minor tweaks to the behavior of the HgRepository.log() helper
function in the integration tests.
Previously this command did not take a revset argument, and instead relied on
the Facebook tweakdefaults extension to use the `--follow` behavior when no
revset was specified. (Without tweakdefaults mercurial uses `tip:0` by
default, which is not what the histedit tests expect.)
I added a revset parameter now, and updated it to default to `::.`. This is
close to the previous behavior, although I intentionally left it reporting
commits from oldest to newest now.
I also updated the log code to add its own delimiter to the template, rather
than requiring callers to always append an escaped nul byte to the template.
Reviewed By: bolinfest
Differential Revision: D5685876
fbshipit-source-id: 01578f62d553be1cd8002b5718d7f12a2f41d4d8
Summary:
Update the integration tests to no longer enable the evolve or fsmonitor
extensions in the test repositories.
evolve has been deprecated at Facebook for a while now and isn't even shipped
as part of our mercurial installation any more. This settings was just causing
a warning to be printed that this extension could not be found.
The fsmonitor extension also didn't have any real effect, even in the backing
repository. We don't create .watchmanconfig files in the test repositories, so
watchman won't watch them. Therefore fsmonitor simply printed warnings that
watchman wasn't watching this repository.
Reviewed By: bolinfest
Differential Revision: D5685879
fbshipit-source-id: 85b8a725bd17890a93be5c71dd5a0f3f1d744598
Summary:
Fix the integration tests to store hg config settings in the .hg/hgrc file in
the backing repository. Previously the tests saved settings to a temporary
file, and then always invoked hg with HGRCPATH pointing at this temporary file.
Unfortunately this resulted in the integration test code using different hg
settings than edenfs, since edenfs was never aware of this temporary file.
Defining the settings in the backing repository's normal .hg/hgrc file means
that edenfs will be able to see these settings as well. The eden post-clone
hooks will also automatically copy these settings in to the mount point, so
that we do not need to use a custom HGRCPATH setting inside the eden mount
either.
Reviewed By: bolinfest
Differential Revision: D5685877
fbshipit-source-id: 1857554d0cf1a585fe55577eb48a87686f9476ca
Summary: This seems a little more user-friendly.
Reviewed By: bradenwatling
Differential Revision: D5686562
fbshipit-source-id: 8142fb9105a3a44823f935fc04187cf0ed2258d7
Summary:
Note that this feature was mostly implemented before this commit, but never
tested. Unsurprisingly, there were bugs.
This change also introduces a new `eden debug hg_copy_map_get_all` subcommand
because that was a straightforward way to verify the internal state of the copy
map on the server side from an integration test.
Adding this test uncovered a key copy/paste bug in `EdenThriftClient.py`
(`hgCopyMapGet` was being invoked instead of `hgCopyMapPut`.)
It also uncovered a bug in `LameThriftClient` because the `compile()` and
`eval()` calls on the output are not appropriate when the return type of the
Thrift endpoint is `string`.
Reviewed By: simpkins
Differential Revision: D5686114
fbshipit-source-id: f0093d2b67062c01982dc5bc1f0db2774b3a9356
Summary:
1.Modified `TreeInode::unloadChildrenNow()` to return number of inodes that have been unloaded.
2.Modified `EdenServiceHandler::unloadInodeForPath()` to return number of inodes that are unloaded.
Reviewed By: simpkins
Differential Revision: D5627539
fbshipit-source-id: 4cdb0433dced6bf101158b9e6f8c35de67d9abbe
Summary:
Added a test case `test_unload_free_inodes_age` to verify the behaviour of unloadChildrenNow with age parameter.
Added new parameter age to `unloadInodeForPath` in eden.thrift, and `EdenServiceHandler`.
Modified `do_unload_inodes` function in `debug.py` to support the new behaviour.
Reviewed By: simpkins
Differential Revision: D5565859
fbshipit-source-id: a35053725be26bc906cf158969cbe21db1cbadde
Summary:
In `hg/eden/__init__.py`, we wrap `match()` in Mercurial's `match.py` in an
attempt to annotate every `basematcher` created in the system with a special
`_eden_match_info` property that we can use in `_eden_walk_helper()` to perform
walks more efficiently. Unfortunately, we missed a case where `scmutil.py`
has a `matchfiles()` function that calls `exact()` in `match.py` directly rather
than going through the generic `match()` function.
This was causing a failure when running `hg revert <filename>` in Eden because
the matcher that was created via `exact()` did not have an `_eden_match_info`.
This commit wraps `exact()` to add the property.
Reviewed By: wez
Differential Revision: D5674660
fbshipit-source-id: 16d1e7648ebd7a23b43b9b1200d3e284e5bc07b0
Summary: Provide a thrift interface to invalidate the cache for an inode denoted by path.
Reviewed By: simpkins
Differential Revision: D5655387
fbshipit-source-id: 887aa4963d216a0d8eed93b6fb8721632cc31d19
Summary:
This is a convenient way to test the `hgGetDirstateTuple()` endpoint in
`eden.thrift`.
Reviewed By: quark-zju, wez
Differential Revision: D5654237
fbshipit-source-id: 0b285e056002d4556733a53293582345f36780b2
Summary: Add a command to lookup the path for an inode given the inode number and eden mount path.
Reviewed By: bolinfest
Differential Revision: D5627411
fbshipit-source-id: 25928f506d3f48d8a6784fe81fb17fa0500d6bc9
Summary:
Updated time stamps of TreeInode accurately on mkdir,rmdir,mknode,symlink,create,unlink and readdir.
updated the `TreeInode::getattr` function to return in-memory timestamps.
Reviewed By: simpkins
Differential Revision: D5568183
fbshipit-source-id: c36f7fb767cd4342aab5cc983eea56e37cd2077e
Summary:
updating atime,ctime,mtime of FileInode on read, write and setattr system calls.
modified `FileInode::stat` function to return accurate inmemory timestamps.
Reviewed By: simpkins
Differential Revision: D5552666
fbshipit-source-id: 86d446f72908663f8db509b7b789d9f35d17df3a
Summary:
We're seeing that this is always set to true for eden,
which is causing buck to run slower than it should.
To make this work correctly, I've augmented our journal data structure
so that it can track create, change and remove events for the various
paths.
I've also plumbed rename events into the journal.
This requires a slightly more complex merge routine, so I've refactored the two
call sites that were merging in slightly different contexts so that they can
now share the same guts of the merge routine. Perhaps slightly
counterintuitive in the merge code is that we merge a record from the past into
the state for now and this is a bit backwards compared to how people think.
I've expanded the eden integration test to check that we don't mix up
create/change/removes for the same path in a given window.
On the watchman side, we use the presence of the filename in the createdPaths
set as a hint that the file is new. In that case we will set the watchman
`ctime` (which is not the state ctime but is really the *created clock time*)
to match the current journal position if the file is new, or leave it set
to 0 if the file is not known to be new. This will cause the `is_new`
flag to be set appropriately by the code in `watchman/query/eval.cpp`;
if the sequence is 0 then it should never be set to true. Otherwise (when
the file was in the `createPaths` set) it will be set to the current journal
position and this will be seen as newer than the `since` constraint on
the query and cause the file to show as `new`.
Reviewed By: bolinfest
Differential Revision: D5608538
fbshipit-source-id: 8d78f7da05e5e53110108aca220c3a97794f8cc2
Summary:
We verify this behavior by adding the appropriate debug statements to our Hg
extension, running `hg update` with the `--debug` flag, and verifying the
expected log statements appear in stdout.
Reviewed By: simpkins
Differential Revision: D5505478
fbshipit-source-id: 389b69c7c1de4fa88fd07fb9eb4dd7e3370e766b
Summary:
This is a relatively simple test that takes two branches with no conflicts and
rebases one on top of the other. It also provides modest checks to ensure Eden
does not load a bunch of inodes unnecessarily when updating to the new head.
This also introduces `EdenServerInspector`, which provides convenience methods
for inspecting the Eden server via Thrift.
Reviewed By: simpkins
Differential Revision: D5504741
fbshipit-source-id: 6636c431658f24a850d0e5404d1a0e4f0528a781
Summary:
Up until now, we have largely ignored Mercurial's logic around transactions.
Previously, the `savebackup()` and `restorebackup()` methods of `eden_dirstate`
were no-ops, but with this change, we provide real implementations for
`savebackup()` and `restorebackup()`.
We no longer try to maintain the `_dirty` field in `eden_dirstate` as of this change
as we should not be exercising any superclass code that relies on its value.
Reviewed By: simpkins
Differential Revision: D5485950
fbshipit-source-id: effa5f83ead8c6eefa613c196d7ec5646da31e4b
Summary:
Added an integration test to verify behavior for `hg rm`. The test passed
without any changes to Eden required!
(Note: this ignores all push blocking failures!)
Reviewed By: simpkins
Differential Revision: D5494421
fbshipit-source-id: ca9578268adb353380e96aced6e2f1caca52b6a3
Summary:
Now `eden_dirstate` can inherit the implementations of the `branch()` and
`setbranch()` methods.
More importantly, we have decided to let `dirstate` do its natural management
of the `.hg` directory (with some exceptions, such as the contents of the
`.hg/dirstate` file) because there are interactions with `localrepo` and other
parts of Mercurial that use the filesystem as the API rather than an explicit
programmatic API, and we want to make sure we maintain the expected invariants.
(Note: this ignores all push blocking failures!)
Reviewed By: simpkins
Differential Revision: D5464059
fbshipit-source-id: 539e142ba11caea3e8487a3cc70d53cc236d2545
Summary:
This updates hg_import_helper.py to override
`mercurial.txnutil._mayhavepending()` to always return True. This allows it
access to commits that are still part of a pending transaction, as long as
`writepending()` has been called on the transaction. (We could have set the
`HG_PENDING` environment variable to the repository root path instead of
monkey-patching `_mayhavepending()`, but this felt more fragile--it requires
normalizing the repository root the same way that mercurial does, and making
sure we get the correct repository root when using the share extension.)
This allows us to remove our override of `commitctx.markcommitted()` in the hg
extension. Previously we replaced this method to defer calling
`dirstate.setparents()` until the transaction was complete. We no longer have
to do this since eden can now access the pending commit.
This should also help fix bugs in the rebase workflow where we previously
failed trying to checkout a still-pending commit.
Reviewed By: wez
Differential Revision: D5345451
fbshipit-source-id: 9c32ab655254c79f030c10b7c9d02563decb04f7
Summary:
This updates the TARGETS files so that we don't run all integration tests in
both the FB and OSS build flavors any more. For the OSS build we only run the
tests in basic_tests.py now.
This reduces the overall time to run all eden tests from about 170 seconds to
about 120 seconds on my server.
Reviewed By: wez
Differential Revision: D5367406
fbshipit-source-id: 81048f415e2f69175059f346e3f81269142a5fc7
Summary:
Add some extra logging to the eden integration to report event timing during
tests. For now I have added some basic logging during set up and tear down.
Test setup is pretty slow for most of our tests. It looks like some of this is
spent waiting for the eden daemon to start up, but a lot of it is just waiting
on hg commands. A lot of tests currently make separate "hg add" calls for each
file they create during repository setup, and this is unfortunately really
slow. Changing these tests to make fewer "hg add" calls would probably speed
things up a fair amount.
Reviewed By: wez
Differential Revision: D5367620
fbshipit-source-id: 99270c1dc4f072b7a0c0eaed7be7730e78b9bb8c
Summary:
Update integration tests to pass logging settings in via the new `--logging`
argument instead of using `--vmodule`. Now that eden log messages have all
been switched from glog to folly logging, the `--logging` flag needs to be
used.
Reviewed By: wez
Differential Revision: D5367618
fbshipit-source-id: 299f73a8376863e46b7fffe59c539e54c78caed4
Summary:
Upstream mercurial recently added calls to dirstate.copies().copy(),
so update the dummy_copymap() class to support this.
Reviewed By: wez
Differential Revision: D5316773
fbshipit-source-id: a2454daf265faaf2bcf1a539c8bf428f8a5e9830
Summary:
1. Moved read, write, mkdir, rm methods in hg/lib/hg_extension_test_base.py to lib/test_case.py.
2. Added integration test case to test unload free inodes.
Reviewed By: simpkins
Differential Revision: D5277870
fbshipit-source-id: b93b6049a10357cf8c92366e6dca3968f7f30c30
Summary:
This is a major change to Eden's Hg extension.
Our initial attempt to implement `edendirstate` was to create a "clean room"
implementation that did not share code with `mercurial/dirstate.py`. This was
helpful in uncovering the subset of the dirstate API that matters for Eden. It
also provided a better safeguard against upstream changes to `dirstate.py` in
Mercurial itself.
In this implementation, the state transition management was mostly done
on the server in `Dirstate.cpp`. We also made a modest attempt to make
`Dirstate.cpp` "SCM-agnostic" such that the same APIs could be used for
Git at some point.
However, as we have tried to support more of the sophisticated functionality
in Mercurial, particularly `hg histedit`, achieving parity between the clean room
implementation and Mercurial's internals has become more challenging.
Ultimately, the clean room implementation is likely the right way to go for Eden,
but for now, we need to prioritize having feature parity with vanilla Hg when
using Eden. Once we have a more complete set of integration tests in place,
we can reimplement Eden's dirstate more aggressively to optimize things.
Fortunately, the [[ https://bitbucket.org/facebook/hg-experimental/src/default/sqldirstate/ | sqldirstate ]]
extension has already demonstrated that it is possible to provide a faithful
dirstate implementation that subclasses the original `dirstate` while using a different
storage mechanism. As such, I used `sqldirstate` as a model when implementing
the new `eden_dirstate` (distinguishing it from our v1 implementation, `edendirstate`).
In particular, `sqldirstate` uses SQL tables as storage for the following private fields
of `dirstate`: `_map`, `_dirs`, `_copymap`, `_filefoldmap`, `_dirfoldmap`. Because
`_filefoldmap` and `_dirfoldmap` exist to deal with case-insensitivity issues, we
do not support them in `eden_dirstate` and add code to ensure the codepaths that
would access them in `dirstate` never get exercised. Similarly, we also implemented
`eden_dirstate` so that it never accesses `_dirs`. (`_dirs` is a multiset of all directories in the
dirstate, which is an O(repo) data structure, so we do not want to maintain it in Eden.
It appears to be primarily used for checking whether a path to a file already exists in
the dirstate as a directory. We can protect against that in more efficient ways.)
That leaves only `_map` and `_copymap` to worry about. `_copymap` contains the set
of files that have been marked "copied" in the current dirstate, so it is fairly small and
can be stored on disk or in memory with little concern. `_map` is a bit trickier because
it is expected to have an entry for every file in the dirstate. In `sqldirstate`, it is stored
across two tables: `files` and `nonnormalfiles`. For Eden, we already represent the data
analogous to the `files` table in RocksDB/the overlay, so we do not need to create a new
equivalent to the `files` table. We do, however, need an equivalent to the `nonnormalfiles`
table, which we store in as Thrift-serialized data in an ordinary file along with the `_copymap`
data.
In our Hg extension, our implementation of `_map` is `eden_dirstate_map`, which is defined
in a Python file of the same name. Our implementation of `_copymap` is `dummy_copymap`,
which is defined in `eden_dirstate.py`. Both of these collections are simple pass-through data
structures that translate their method calls to Thrift server calls. I expect we will want to
optimize this in the future via some client-side caching, as well as creating batch APIs for talking
to the server via Thrift.
One advantage of this new implementation is that it enables us to delete
`eden/hg/eden/overrides.py`, which overrode the entry points for `hg add` and `hg remove`.
Between the recent implementation of `dirstate.walk()` for Eden and this switch
to the real dirstate, we can now use the default implementation of `hg add` and `hg remove`
(although we have to play some tricks, like in the implementation of `eden_dirstate.status()`
in order to make `hg remove` work).
In the course of doing this revision, I discovered that I had to make a minor fix to
`EdenMatchInfo.make_glob_list()` because `hg add foo` was being treated as
`hg add foo/**/*` even when `foo` was just a file (as opposed to a directory), in which
case the glob was not matching `foo`!
I also had to do some work in `eden_dirstate.status()` in which the `match` argument
was previously largely ignored. It turns out that `dirstate.py` uses `status()` for a number
of things with the `match` specified as a filter, so the output of `status()` must be filtered
by `match` accordingly. Ultimately, this seems like work that would be better done on the
server, but for simplicity, we're just going to do it in Python, for now.
For the reasons explained above, this revision deletes a lot of code `Dirstate.cpp`.
As such, `DirstateTest.cpp` does not seem worth refactoring, though the scenarios it was
testing should probably be converted to integration tests. At a high level, the role of
`DirstatePersistence` has not changed, but the exact data it writes is much different.
Its corresponding unit test is also disabled, for now.
Note that this revision does not change the name of the file where "dirstate data" is written
(this is defined as `kDirstateFile` in `ClientConfig.cpp`), so we should blow away any existing
instances of this file once this change lands. (It is still early enough in the project that it does
not seem worth the overhead of a proper migration.)
The true test of the success of this new approach is the ease with which we can write more
integration tests for things like `hg histedit` and `hg graft`. Ideally, these should require very
few changes to `eden_dirstate.py`.
Reviewed By: simpkins
Differential Revision: D5071778
fbshipit-source-id: e8fec4d393035d80f36516ac050cad025dc3ba31
Summary:
I was working on a new test and I got an error that `directaccess` must be
enabled for `inhibit` to work.
Reviewed By: simpkins
Differential Revision: D5077133
fbshipit-source-id: cc5235c845e3f299f96e1c901ef4aea18ca57b76
Summary:
I had to add simple implementations to various things in `edendirstate`
in order to be able to run `hg histedit`. There is still a lot more to do, but
at least this gives us a starting point to iterate and a test to demonstrate
the most simple functionality.
Reviewed By: wez
Differential Revision: D5049308
fbshipit-source-id: 34727f633c003cacae44108eb3ece06590098c7b
Summary:
Note that we must specify quite a few extensions to get behavior that is
representative of how Hg works at Facebook.
Reviewed By: DurhamG
Differential Revision: D5057478
fbshipit-source-id: ee774a9b8dcebe82e4b19cc52f9b0b5a53e6420c
Summary:
Recall that we override `$HOME` in integration tests, so this will not overwrite
your personal `~/.hgrc` when you run an integration test.
An upcoming integration test for `hg histedit` that I am working on requires
this value to be set.
Reviewed By: wez
Differential Revision: D5051112
fbshipit-source-id: 2fd8541aa6504640b08337fdc22160e243beaae3
Summary:
`HgExtensionTestBase.assert_status()` was added in D4814422, but it was only
applied to `update_test.py`. This change updates the docstring (it appears to
have been copy/pasted from a nearby method), and makes use of it in the other
integration tests.
Reviewed By: wez
Differential Revision: D5050775
fbshipit-source-id: bb70740b6f455a84e7a22c3286c8ddbe2462f816
Summary:
Previous to this change, when I would add `import pudb; pudb.set_trace()` to do
some debugging, two annoying things would happen:
- I would have to edit the `TARGETS` file to add `pudb` as a dependency and
then `buck build eden/integration/hg` again.
- When I hit a breakpoint using `pudb`, I would have to go through the welcome
screen, change the theme, etc., because my settings were not found.
Now that I figured out what the problem was, I added instructions to the
`TARGETS` file to help others fall into the pit of success.
Reviewed By: wez
Differential Revision: D5050725
fbshipit-source-id: 1896f9f52eb056b3295b2d8e896dabb5d990ba22
Summary:
This fixes "hg commit" so that it correctly updates the in-memory snapshot.
This has been broken ever since I added the in-memory snapshot when
implementing checkout(). The existing scmMarkCommitted() method updated only
the Dirstate object and the on-disk SNAPSHOT file.
This diff fixes checkout() and resetCommit() to clear the Dirstate user
directives correctly, and then replaces calls to scmMarkCommitted() with
resetCommit().
Reviewed By: bolinfest
Differential Revision: D4935943
fbshipit-source-id: 5ffcfd5db99f30c730ede202c5e013afa682bac9
Summary:
This updates how we build and package the eden hg extension, and how we find it
during integration tests.
- Update the extension to always look relative to its current location to find
the other modules it depends on. This ensures that the integration tests
always find modules from the local repository, and do not use the modules
installed on the system.
- Add a buck rule to unpack the python archive at build time. This is needed
for integration tests to use the local version of the module.
- Ensure that we install a correct `hgext3rd/__init__.py` module in the eden
extension directory. This is required to correctly set up `hgext3rd` as a
namespace package. This also unfortunately needs to be a `.py` file, and not
just a .pyc file. (The pkgutil.expand_path() code looks specifically for
directories containing `__init__.py` files, and does not check for
`__init__.pyc`.)
- Update the extension to only try importing the native thrift modules if we
are running python 2.7.6 or greater. Python 2.7.6 is the first that supports
unicode arguments to `struct.pack()`, which thrift requires. Python 2.7.5 can
import the thrift modules, but throws errors when trying to run them.
Reviewed By: bolinfest
Differential Revision: D4935279
fbshipit-source-id: 9af81736124c55476a5eb5beba9474a4371a639b
Summary:
Move the code for the command-line tool up one directory, out of eden/fs.
This better separates the code so that eden/fs contains code for the edenfs
daemon, while eden/cli contains code for the command line tool.
Reviewed By: bolinfest
Differential Revision: D4888633
fbshipit-source-id: 5041e292c5353d05122eefe5db3257289e31239a
Summary:
Fix a subtle crash during checkout when handling newly added entries that
already exist in the working directory: CheckoutAction passed the entry name to
checkoutUpdateEntry() as a PathComponentPiece. However, this
PathComponentPiece could refer to the entry name owned by newScmEntry_, and it
also passed newScmEntry_ into checkoutUpdateEntry() as an rvalue reference.
As a result, if the string data was stored invalidated by the move the name
would no longer be valid when checkoutUpdateEntry() tried to use it.
This bug is triggered by doing an "hg update --clean", where a file added in
the destination commit already exists on disk, and has an entry name of 23
characters or less. (The 23 character limit is fbstring's upper bound on
small string optimizations, where it will store the string data inline in the
object, causing it to be invalidated on move.)
This also fixes a crash in a VLOG() statement when the verbose log level for
TreeInode.cpp was set to 4 or greater.
Reviewed By: bolinfest
Differential Revision: D4882544
fbshipit-source-id: 917ede6eeae2224aaa0724b8b30324f3c3a5c924
Summary:
Update the hg extension to implement dirstate.rebuild(). This is necessary for
the `hg reset` command. This also now implements dirstate.setparents() for
cases when there is only one parent.
Reviewed By: wez
Differential Revision: D4823780
fbshipit-source-id: 802de006e03860995095dc3af17acb2eb05f4e8b
Summary:
The intent is to provide a way to locate the SNAPSHOT file
for tools that want to have a very fast way to figure out the commit
id without making any RPCs or subprocess invocations.
Reviewed By: simpkins
Differential Revision: D4824176
fbshipit-source-id: 5adca225d9984146852dad1e83de0d903848c1e5
Summary:
This diff fixes FileData::stat() so that we report reasonable timestamp values
on non-materialized files, rather than always leaving them as 0. We set the
timestamps to the time that we created the FileInode. This ensures that
timestamps are updated correctly when files are modified by a checkout
operation.
Note that for materialized files the code reports the timestamp of the overlay
file. This diff does not modify that behavior. However, this behavior is
incorrect, as the overlay file timestamps are not updated by a FUSE client
opening, modifying, then closing a file (since we keep the underyling overlay
file handle open, and don't close it).
In the future we'll need to implement our own tracking of atime, mtime, and
ctime values. We should probably store these in a header inside the overlay
file. For now, this diff is a stop-gap measure that ensures we at least update
non-materialized file timestamps correctly on checkouts.
Reviewed By: bolinfest
Differential Revision: D4765632
fbshipit-source-id: 478da6441e213cdfe830f1c5129212182ce4eeb0
Summary:
Update the integration test code to allow TestCase classes to control the
--vmodule settings used when starting the edenfs daemon.
D3851805 initially set the vlog level for RequestData to 5 to help debug
issues. However, this log level doesn't really make sense for most of the
mercurial integration tests: most mercurial commands checks for the existence
of lots of files, causing the logs to be filled with useless ENOENT messages
when the RequestData log level is this high.
Reviewed By: wez
Differential Revision: D4814402
fbshipit-source-id: 1127c0a25b656ea1a710ca54a59a9407d66a5659
Summary:
Add an assert_status() method to the hg integration tests that runs "hg
status", parses the output, then compares it to expected results.
Reviewed By: wez
Differential Revision: D4814422
fbshipit-source-id: 24ebdc2e0239c4833953c31e5786cc320bcd9d62
Summary:
Update HgRepository.hg() to catch subprocess.CalledProcessError exceptions and
translate them into subclass that returns a more helpful message.
The CalledProcessError includes the command stderr as a member variable, but
unfortunately does not include this in the output from `__str__()`. The stderr
output is usually important for helping debug test failures.
Reviewed By: wez
Differential Revision: D4814401
fbshipit-source-id: 4f010bb3ce33833c55d18768997d2d1bb6a001b8
Summary:
The hg_import_helper script that eden uses to import data from mercurial keeps
a long-lived repository object open. This caches some data about the
repository, and if new commits are added after it was created, it can fail to
see them.
This updates hg_import_helper.py to catch errors that occur when trying to use
the repository objects. The code will invalidate the repository object and
then retry the operation once, in the hopes that it will now succeed after
invalidation.
Reviewed By: bolinfest
Differential Revision: D4752659
fbshipit-source-id: 1c75c84766d6bbda0710882a338eaa09e0cb0030
Summary:
The kernel can return ENOENT in response to invalidation notification if we
have never told the kernel about the inode in question. This resulted in
spurious errors during checkout when updating files that were loaded internally
by edenfs rather than via FUSE call. For instance, this was commonly triggered
by .gitignore files, which eden loads on its own to perform ignore processing.
Reviewed By: bolinfest
Differential Revision: D4752630
fbshipit-source-id: d4e092643a8d33cf33709f7e3664289f167ac093
Summary:
I found it rather awkward in HgExtensionTestBase that self.repo is not actually
the repository being tested. It was instead the repostiory used as the backing
store for the mercurial data, and self.repo_for_mount was the repository being
tested.
This diff renames the two repository classes, so that self.backing_repo is now
the backing store repository, and self.repo is the repository being tested.
In order to do this I changed HgExtensionTestBase to derive directly from
EdenTestCase. Previously it derived from EdenHgTest, and was letting
EdenHgTest set up self.repo. It seemed more understandable to avoid deriving
EdenHgTest now since self.repo is not the repository that needs to be set up
initially.
Reviewed By: bolinfest
Differential Revision: D4752631
fbshipit-source-id: d8b542b0ecead66b965af1a582085345e28b2908
Summary:
Previously the eden hg extension short-circuited the checkout operation if the
destination commit was the same as the one currently checked out. This was
incorrect if --clean was specified, since we do need to reset the working
directory state in this case.
This updates the extension code to always make the thrift checkout() call when
doing a force checkout.
This also avoids calling applyupdates() to resolve conflicts when force=True.
When doing a force checkout, eden reports files with conflicts that it
overwrote, but these do not need to be resolved by mercurial.
Additionally, this also updates a couple other APIs that have recently been
changed in upstream mercurial: merge.update() now takes an updatecheck
argument, and repo.join() should now be written as repo.vfs.join().
Reviewed By: bolinfest
Differential Revision: D4752510
fbshipit-source-id: e1ee92d086315e35a1378f674e668876a667c0ce
Summary:
this is the bare minimum to support creating unix domain sockets.
We only support using mknod to create a unix socket; other uses will yield an error.
I've added an rdev field as a sibling of the existing mode field that we track,
as that is the additional parameter that we need to track as part of the
special file node.
Special file nodes are tracked in the overlay as empty files.
Reviewed By: bolinfest
Differential Revision: D4774099
fbshipit-source-id: 0824b7e509063faa8bede7aff82a7c51930c4f83
Summary:
It's not really magic because we don't have a virtual directory
inode base any more. Instead, we mkdir and populate it at mount time.
What is slightly magical about it is that we give it some special powers:
* We know the inode number of the eden dir and prevent unlink operations
on it or inside it.
* The .eden dir is present in the contents of the root inode and will
show up when that directory is `readdir`'d
* When resolving a child of a TreeInode by name, we know to return the
magic `.eden` inode number. This means that it is possible to `stat`
and consume the `.eden` directory from any directory inside the eden
mount, even though it won't show up in `readdir` for those child dirs.
The contents of the `.eden` dir are:
* `socket` - a symlink back to the unix domain socket that our thrift
server is listening on. This means that it is a simple
`readlink(".eden/socket")` operation to discover both whether a directory
is part of an eden mount and how to talk to the server.
* `root` - a symlink back to the root of this eden mount. This allows
using `readlink(".eden/root")` as a simple 1-step operation to find
the root of an eden mount, and avoids needing to walk up directory
by directory as is the common pattern for locating `.hg` or `.git`
dirs.
Reviewed By: simpkins
Differential Revision: D4637285
fbshipit-source-id: 0eabf98b29144acccef5c83bd367493399dc55bb
Summary:
The privhelper code was erasing elements from a std::unordered_map before it
was done using the iterator pointing to that element. This causes memory
corruption issues.
Between this and some of my other recent unmount fixes (D4548030, D4547938)
this makes the bind-mount-related integration tests work.
Reviewed By: bolinfest
Differential Revision: D4727850
fbshipit-source-id: 6d1fda3f89cb91c89d0020921b1805fc10e65785
Summary:
I originally added this to facilitate `hg status` but we didn't end up
using it. Rather than keeping it around and updating the tests in the `.eden`
dir diffs that follow, let's just remove it.
Reviewed By: simpkins
Differential Revision: D4610768
fbshipit-source-id: 158c0207f88980e86aeeddf75e6fd49763d2a402
Summary:
Move the integration tests from eden/fs/integration up one directory, to
eden/integration.
The main benefit is that this makes it easy to run just the edenfs unit tests
by running "buck test eden/fs/...". These unit tests complete much more
quickly than the full set of integration tests, providing a faster test suite
to re-run repeatedly during development. The integration tests can be run with
"buck test eden/integration/...", and the full set of tests can still be run
with "buck test eden/..."
Reviewed By: wez
Differential Revision: D4490247
fbshipit-source-id: 5ceb5a19526f56e1cb926f352fa30ad2f1212c05
