Summary:
The takeover data for a fully-loaded monorepo was 20 MB. Two
monorepos was 40 MB, exceeding the previous 32 MiB limit and causing
takeover to fail. Bump to 512 MiB to minimize the chance of
allocating too much given bogus data while still supporting large
numbers of checkouts.
Reviewed By: simpkins
Differential Revision: D7661637
fbshipit-source-id: 1dc54055e7ac3ce0f6cdbf4a7e5d372c2674dfee
Summary: Temporary fix to bump the takeover timeout to five minutes
Reviewed By: simpkins
Differential Revision: D7660642
fbshipit-source-id: 5b3a0347aa1443e395e9461d33459c62ffce5b2d
Summary:
UnixSocket traverses the IOBuf chain twice. Refactor that
into a common function because the next diff caps the size of
individual iovecs.
Reviewed By: simpkins
Differential Revision: D7659062
fbshipit-source-id: 88b7d63669d8189b96434c38a6e499ed3b5ebbe6
Summary:
fdatasync() has a substantial cost and is unnecessary given
our durability guarantees (and the fact that most filesystems on Linux
try to avoid data loss in this common write + rename situation anyway)
Reviewed By: simpkins
Differential Revision: D7641131
fbshipit-source-id: d041e7090dc05a4d4400f86cad9501aa8a6988a9
Summary:
Fix a couple issues when when performing an `hg update` with updatecheck set to
`noconflict` and when conflicts are detected:
- Report the files with conflicts. The conflicts may be non-obvious to the
user if the conflicts are in ignored files that are not tracked in the
current commit but are present in the destination commit.
- Do not invoke the preupdate hook or create the .hg/updatestate file until
after we have performed the conflict check. Otherwise the repository will be
marked as in the middle of an unfinished update even though we never
attempted the update itself.
Reviewed By: quark-zju
Differential Revision: D7581782
fbshipit-source-id: e290ebd6a27f228a805bea40aabb3f3cc633cc7d
Summary:
Update the assert_status() function in the integration tests so that it also
checks for unfinished update/rebase/graft/etc operations. We unfortunately
have to manually check for the presence of these files ourselves, since
`hg status` provides no mechanism to report this data when `HGPLAIN` is set.
Reviewed By: quark-zju
Differential Revision: D7581781
fbshipit-source-id: 230234e5b8ce28cd3569cdacced686fed2a9dd32
Summary:
This rewrites the `walk()` and `status()` methods in the `eden_dirstate` class.
The changes to the `status()` function should not have any major behavior
changes: this primarily moves logic out of `EdenThriftClient.getStatus()` and
into `dirstate.status()`
The changes to the `walk()` function do fix a number of bugs. We now implement
walk by combining Eden's `getScmStatus()` results with the dirstate's
non-normal files and the current parent commit's manifest. All of the glob
matching is done purely in python now. This fixes some cases where Eden's glob
handling behavior did not support some glob patterns allowed by mercurial. The
Eden glob() call also always returned ignored files and hidden files (such as
everything inside the `.hg` directory). I believe this behavior is desired by
watchman, but it caused problems for the `walk()` code.
Reviewed By: chadaustin
Differential Revision: D7565503
fbshipit-source-id: deb91b4772501e8fbdba56e5c099a72d55fb61ee
Summary:
Update `eden_dirstate.status()` to invoke the `match.bad()` and
`match.explicitdir()` appropriately when any of the explicitly specified match
paths either don't exist or refer to directories.
Reviewed By: chadaustin
Differential Revision: D7556850
fbshipit-source-id: 2e8d88ff0e3c09cea2f3ef5b18c9a214231815ab
Summary:
Drop the log level of the "no dir data for inode" message from DBG2 to DBG3 so
we do not have it enabled by default. (Our default log settings are
eden=DBG2.)
Reviewed By: chadaustin
Differential Revision: D7620918
fbshipit-source-id: 25cf30af08054f3ba879bd57f3841b1035e0c75e
Summary:
Implement `dirstate.matches()` in Eden by calling `matches()` on the parent
commit's context, and then updating that result with the list of non-normal
files from Eden's dirstate.
Reviewed By: chadaustin
Differential Revision: D7556847
fbshipit-source-id: 34e78c0c125cd1b184a7773a149edafdca34fee1
Summary:
Add integration tests to check the behavior of the `hg journal` command in an
Eden checkout.
Reviewed By: wez
Differential Revision: D7512339
fbshipit-source-id: 9a83a1752835bc6ded9f0c3b2cb29ec24a6db631
Summary:
When running the integration tests, set HGRCPATH to point to the configs found
at `scm/hg/fb/staticfiles/etc/mercurial`
This does affect mercurial's behavior in some cases, and therefore required
changes to some of our tests:
- `hg diff` output now uses an output style similar to `git diff`
- merge conflict markers include some extra commit metadata
Reviewed By: wez
Differential Revision: D7512321
fbshipit-source-id: 6141605147797ea8b13fdb5a06f67beaeff6afe7
Summary:
This slightly refactors the way that EdenTestCase and EdenRepoTest initialize
repositories. This removes the `create_repo()` method from EdenTestCase and
replaces it with separate `create_hg_repo()` and `create_git_repo()` methods.
The `EdenRepoTest` subclasses now provide alternate `create_repo()`
implementations instead of `get_repo_class()`.
This cleans up the code a bit, since `create_repo()` no longer takes different
arguments based on what type of repository is being created. This also will
make it easier in upcoming diffs to further customize the logic that occurs in
`create_hg_repo()`.
Reviewed By: chadaustin
Differential Revision: D7512320
fbshipit-source-id: d268b0ac0ffb33f3dfd34f2dd1917d57033c81aa
Summary:
Every day I read this document and notice more of the awkward bits. :)
(Note: this ignores all push blocking failures!)
Reviewed By: pkaush
Differential Revision: D7602548
fbshipit-source-id: 71e30de869d97411e8fb86de0701ca7890942ddf
Summary:
I'm about to change the code paths through which directories
are saved to and loaded from persistent storage. To help, consolidate
those operations.
Reviewed By: simpkins
Differential Revision: D7597978
fbshipit-source-id: 9cd20027746b0a372a97b7f5daf07c1b54e6b9ce
Summary:
Add a document that begins to sketch out the various transitions that
our inode data structures make so we can reason about their
correctness
Reviewed By: simpkins
Differential Revision: D7434693
fbshipit-source-id: 5478c108c338ccdadc22e864b077195c8be7d1b7
Summary:
Once a mount point has been unmounted we no longer need to care about
outstanding FUSE reference counts--we can treat them as if they are all zero.
This updates EdenMount to tell the InodeMap when the mount point is unloaded,
and changes InodeMap::unloadInode() to make use of this information when
deciding if it needs to remember the inode information.
Previously InodeMap would save information for inodes with outstanding FUSE
reference counts. Writing all of this state to the overlay could take a
non-trivial amount of time.
Reviewed By: chadaustin
Differential Revision: D7555998
fbshipit-source-id: 0896f867ce850ab3e61c262776d536de003685ff
Summary:
Remove the EDEN_HAS_COMMON_STATS checks now that the common/stats stubs have
the required APIs needed by Eden.
Reviewed By: wez
Differential Revision: D7479593
fbshipit-source-id: cc3db50288bfea7aefd6c91391ab800628b7978f
Summary:
Add a macro to help users define the `getBaseLoggingConfig()` function.
While I would prefer to avoid macros if possible, this seems worthwhile. This
saves 4 or 5 lines of boilerplate code in each program that sets a custom base
logger setting. It also reduces the likelihood of a developer accidentally
having a typo in the function name, which would still build successfully but
not have the desired results.
Reviewed By: chadaustin
Differential Revision: D7457652
fbshipit-source-id: 1c316c7ea6949c16bd7b61c0440cc1ee69ecb83e
Summary:
Make several performance improvements to INSTRUMENT_THRIFT_CALL:
- Avoid heap-allocating the ThriftLogHelper objects.
- Avoid evaluating the log arguments if the log level is not enabled.
Reviewed By: chadaustin
Differential Revision: D7556846
fbshipit-source-id: e111e24e44499c5cf9725ded2b958a7dcb2c3e26
Summary:
Add a helper function to make sure we log commit hash arguments as hexadecimal
rather than binary data. This is similar to `hashFromThrift()`, except that it
always returns a string result and does not throw on invalid input.
Reviewed By: chadaustin
Differential Revision: D7556916
fbshipit-source-id: 87422de3f178700d378f5ddc45172efd38a13799
Summary:
Rename `getRefcount()` to `debugGetFuseRefcount()` to make it clear that this
returns the FUSE reference count rather than the pointer refcount, and that
the result should not be used for anything other than debugging or diagnostic
purposes.
There is already a `getFuseRefcount()` method to check the FUSE refcount while
the inode is being unloaded. The only time it is safe to check the FUSE
refcount for programmatic purposes, and `getFuseRefcount()` has a `DCHECK()` to
help ensure that this is the only time it is called.
(`getFuseRefcount()` also predates `getRefcount()`, which was added for the
`eden debug inode` command in D5277870.)
Reviewed By: chadaustin
Differential Revision: D7555997
fbshipit-source-id: e66f2b3d2b9eb14f8f8878d78978851929198a31
Summary:
Change EdenMount to destroy the Overlay object as soon as the `shutdown()`
operation completes. Previously the Overlay would not get destroyed until
EdenMount::destroy() was called.
During graceful restart we transfer mount information to the new process after
`shutdown()` completes, but potentially before `destroy()` has finished. This
previously resulted in a race condition where the new process could start
opening the overlay before the old process had released the lock. This change
should fix that race condition.
Reviewed By: chadaustin
Differential Revision: D7543295
fbshipit-source-id: 82ae33fe6bb0aa9f0a3b010fdd1d350c63442420
Summary: Each Overlay object should be owned only by its EdenMount.
Reviewed By: chadaustin
Differential Revision: D7543294
fbshipit-source-id: 6db40fea31ce298c61d047cba9165887e32926a1
Summary:
A couple have users have run into a bug where mercurial's `ctx.manifestnode()`
function returns an empty string. I believe treemanifest was been enabled both
times where this was seen. We did see that restarting eden (which restarts the
hg import helper) made the problem go away.
This updates the import helper to throw an exception if `ctx.manifestnode()`
ever returns the empty string. We will invalidate the cache and retry if this
occurs.
Reviewed By: quark-zju
Differential Revision: D7542922
fbshipit-source-id: 42a31835a65217473bf252b10f9cf82dd901f89b
Summary:
Change getScmStatus() so that callers must explicitly specify the commit to
diff against. This should help avoid race conditions around commit or checkout
operations where the parent commit has just changed and eden returns status
information against a commit that wasn't what the client was expecting.
This should still maintain backwards compatibility with older clients that do
not send this parameter yet: we will simply receive the hash as an empty string
in this case, and we still provide the old behavior in this case.
Reviewed By: wez
Differential Revision: D7512338
fbshipit-source-id: 1fb4645dda13b9108c66c2daaa802ea3445ac5f2
Summary:
Update the GitRepository and HgRepository classes in the integration test code
to use the find_executable module to find the location of git and hg.
Reviewed By: ryanmce
Differential Revision: D7512037
fbshipit-source-id: 139072c3493f70abd0dc92acab295d707bee8ad6
Summary:
Refactor the find_executables module to only look up executables when they are
needed, instead of performing all look-ups immediately even if the test in
question may not need all of the binaries. Also add a _find_exe() helper
function to eliminate some code duplication.
Reviewed By: ryanmce
Differential Revision: D7512038
fbshipit-source-id: fdfb8ec70b3f6292603826b3fb22c01dbd1f0d72
Summary:
Add an integration test that nonmaterialized files have the same inode
numbers after restarting the Eden mount.
Reviewed By: simpkins
Differential Revision: D7328311
fbshipit-source-id: 989abbed38626839e0badffeac2d86dc46c8dd72
Summary:
Now that D7451330 writes saved inode numbers into the overlay, we
need to make sure they get deleted when checking out between trees.
Reviewed By: simpkins
Differential Revision: D7327942
fbshipit-source-id: 9593c7abe9d2e424b9ca1d9c5a5ab8b285867e6e
Summary:
Always remember the inode numbers of children when unloading
a tree. This fixes a bug where, if a tree is unloaded during a build,
child inode numbers would be regenerated, causing the same header to
be compiled once. gcc and clang use inode numbers to decide whether
to #include a file that uses #pragma once, so they must remain
consistent while a mount is up.
This is perhaps a bit controversial because it saves _every_ tree,
unconditionally, to the overlay. I will either stack another diff
that ensures these overlay trees are deleted when checking out to a
new tree or make that change in place here, depending on how urgently
this fix is needed.
Reviewed By: simpkins
Differential Revision: D7451330
fbshipit-source-id: 1c3bc0d55327924c88d1a559d2557cfc158991f2
Summary:
Split up unmountAll() into two separate methods for the takeover vs
non-takeover code paths. The logic for these two situations is fairly
different, and they have different return types. Keeping them separate makes
it a bit easier to understand these two separate code paths.
In the future we may also want to update this code further so it can transfer
mounts to the new process as soon as each mount is done shutting down, which
would make these code paths diverge further.
Reviewed By: chadaustin
Differential Revision: D7504363
fbshipit-source-id: 50d19b9a660f791b69f3f5f99f3943040e6af368
Summary:
We already had type annotations on most of the `hg` integration tests. This
adds them for the top-level (non-source-control-specific) tests.
typeseverywhere
Reviewed By: wez
Differential Revision: D7459281
fbshipit-source-id: 41266b232ded510d6b63dd3e62c272a0cd6a0e1a
Summary:
Implement the tear-down actions using addCleanup() rather than through a
tearDown() function. This simplifies the logic, and also makes it easier to
write the code that can be type-checked more easily. (We no longer need to
initialize some variables to `None` just to help `tearDown()` identify if they
have been set up yet or not.)
Reviewed By: wez
Differential Revision: D7459278
fbshipit-source-id: de32ebfcb5474a9583f2a51cc9277102189cb77e
Summary:
Update the eden_repo_test decorator so that it no longer automatically adds
`EdenRepoTestBase` as a parent class. Individual test classes still specify
`EdenRepoTest` as their parent now.
This enables `mypy` to correctly figure out that the individual test classes
derive from `unittest.TestCase`.
This basically does the same thing as D6268258 for the top-level integration
tests.
Reviewed By: wez
Differential Revision: D7459280
fbshipit-source-id: 5d18bd241dad77d55541ac3fa1d169496ffe7003
Summary: Materialized entries are stored with an unset hash. Fix the CLI.
Reviewed By: simpkins
Differential Revision: D7465192
fbshipit-source-id: 41e50c7214feb8afb1f3bf47252e4538945a5737
Summary:
My muscle memory doesn't quite know where the edenfs log file
is, so add a shortcut command to view it.
Reviewed By: simpkins
Differential Revision: D7418743
fbshipit-source-id: 290b51b63fc3573663ad53f44e50dfd073c211c8
Summary: Add test demonstrating inode numbers are not consistent across takeover.
Reviewed By: simpkins
Differential Revision: D7300867
fbshipit-source-id: 1e142c28ac00b569b96b5a1867bc8632771c2be9
Summary:
This integration tests appears to be frequently hanging on sandcastle, and
leaving behind unkillable zombie processes. I'm disabling it for now until we
can figure out why it hangs and fix it and/or make sandcastle smart enough to
be able to successfully tear down the FUSE mounts.
These stuck processes are likely stuck waiting on a kernel lock that can only
be released by using `/sys/fs/fuse/connections/DEVICE_ID/abort`
Reviewed By: wez
Differential Revision: D7440282
fbshipit-source-id: 70034fd65267eff18bf5a052cde8d11e0a66b46b
Summary:
Update the Eden CLI to preserve the `ASAN_OPTIONS` environment variable when
starting edenfs.
Reviewed By: wez
Differential Revision: D7440148
fbshipit-source-id: c9932c5c14b5a2e040309268f95229c04715bb28
Summary:
Fix FuseChannel::processSession() to always process all FUSE requests that it
reads. Previously it checked to see if it should stop immediately after
reading FUSE request. It was possible for the old process to successfully read
a FUSE request, see that it was supposed to stop, and then exit this worker
thread without ever processing this FUSE request. This would cause the client
that sent the request to hang indefinitely, since no response would ever be
sent.
Reviewed By: wez
Differential Revision: D7436867
fbshipit-source-id: c58c2f6c49102fa6b66ac83fc1639595a5277ce0
Summary:
Perform cleanup with an `addCleanup()` function rather than in `tearDown()`.
This is slightly simpler, but also avoids throwing an exception in `tearDown()`
if `setUp()` failed before it created the thrift client.
Reviewed By: wez
Differential Revision: D7436636
fbshipit-source-id: fe525c1fac86bb866c560cca0ad940ea87d96f07
Summary:
Update the `eden clone` command to accept extra command line flags telling it
where to find the edenfs executable, as well as additional arguments to pass
to edenfs.
This enables the `test_clone_should_start_daemon()` integration test to pass in
flags to tell it how to start edenfs correctly.
Reviewed By: chadaustin
Differential Revision: D7433367
fbshipit-source-id: 0eddd93d1332e113fb85fa4b8fc87ba51d7eab2c
Summary:
Update most of the Eden CLI code to include python type annotations.
I believe the stats.py and stats_print.py are the only CLI files that do not
have complete typing information now.
Reviewed By: chadaustin
Differential Revision: D7433368
fbshipit-source-id: dfd6a064cacffeeed9147739da7064f3303de789
Summary:
Update test_version() so that it can succeed even if the RPM is not installed
on the local system.
Reviewed By: wez
Differential Revision: D7433366
fbshipit-source-id: 5502cca02f9b481cc136beb665048525bae72807
Summary:
This cleans up the two createOverlayFile() implementations as well as
saveOverlayDir() to all use the same logic for writing out overlay files. This
new logic is more efficient than some of the old code, as we only open the file
once.
Additionally, this also changes the Overlay code to use openat() and related
APIs when accessing files in the overlay, so that we can specify paths relative
the top-level overlay directory. This means that all pathnames are guaranteed
to fit in a small path length known at compile time, and we can avoid ever
allocating path names on the heap.
One potential downside of using openat() is that this functionality may not be
available on Mac OS X, so we will likely still need to provide alternate
implementations that do use heap-allocated absolute paths for Mac support.
Reviewed By: chadaustin
Differential Revision: D7411499
fbshipit-source-id: dd76395130dda4c2b9403cce04f4201f6def0d10
Summary: Add a test case to confirm the behavior of Overlay::getFilePath().
Reviewed By: chadaustin
Differential Revision: D7411498
fbshipit-source-id: f859f29e252c1c296eb236113ad2a8b0a892db37
Summary:
Most of the logic in TreeInode::create(), symlink() and mknod() was very
similar. This adds a TreeInode::createImpl() helper method that these three
functions can share.
This also updates TreeInode::create() to throw EEXIST if the file in question
does exist, rather than using EDEN_BUG(). I believe it is possible (but
unlikely) for this to occur in practice since inode invalidation operations
triggered by a checkout are now processed asynchronously.
Reviewed By: chadaustin
Differential Revision: D7411500
fbshipit-source-id: 85d97995139eee6bff96381561fc28e76d7a2b7c
Summary:
Update all Overlay APIs to accept InodeNumbers and compute the path to the
overlay file internally, rather than requiring the caller to pass in the path
to the file inside the overlay.
Reviewed By: chadaustin
Differential Revision: D7411497
fbshipit-source-id: d057babb0b1b8d6c3b9adf1a21b84eed363dffa1
