Summary:
Sadly, we didn't derive as much value from these as we would
have liked, and now they are a source of flakeyness in our tests.
Reviewed By: genevievehelsel
Differential Revision: D19460792
fbshipit-source-id: 48c82cc2d1fdbd81bece4057e93799fbcc4f4725
Summary:
Instead of clearing every single cached object when the total size
exceeds the ephemeral storage limit, keep a limit per object type and
only clear those that exceed their quota.
Reviewed By: simpkins
Differential Revision: D19358312
fbshipit-source-id: 6918d6f4cc2931aed79a9025d0e0f357ede515e0
Summary: adds a cli debug command to inspect the working copy parent. by default just returns eden's snapshot contents, but adds optional --hg flag to print Mercurial's dirstate information
Reviewed By: chadaustin
Differential Revision: D19167518
fbshipit-source-id: b65e112df6abe4e0e7a8a528a90b2e3d17297e66
Summary:
Remove some half-baked, unnecessary logic for caching sizes separately
from SHA-1. Eden's backing stores do not support chunking large files
yet, so there's no value in caching content SHA-1 and size
separately. This fixes a scenario where fetching blob size and then
SHA-1 would result in two backing store imports.
Reviewed By: fanzeyi
Differential Revision: D19169096
fbshipit-source-id: dc32f3313e5f4230c06a5bbaa67da7bf0febaba8
Summary: There is one instace of `getScmStatusBetweenRevisions` in use - it is used in the eden cli in a hacky way to check if a commit hash is valid. Since this is not used anywhere else in a meaningful way, this replaces that use case with a hg call and depreciates `getScmStatusBetweenRevisions`
Reviewed By: simpkins
Differential Revision: D18690026
fbshipit-source-id: 02bd2c20a0f631ec41116f9fd4e18d14369298ef
Summary:
A spike in automatic GCs usually implies something has gone wrong. Log
an event for each one, recording the cache size prior to the GC and
the cache size after.
Reviewed By: simpkins
Differential Revision: D18902580
fbshipit-source-id: 158b2635733a415a9fcc7c412b2c0f44ed04aa01
Summary:
Two bugs conspired to cause edenfs after a graceful restart to think
the kernel supported FUSE_NO_OPENDIR_SUPPORT when it didn't: the
connection info struct wasn't zeroed, and FUSE connection capabilities
weren't properly mirrored into the Dispatcher upon graceful
restart. Fix both and add an integration test.
Reviewed By: simpkins
Differential Revision: D18903761
fbshipit-source-id: 23f4db3e240ee7d035f707820072c606a45f1138
Summary: This updates the hg and telemetry wrapper callsites of getScmStatus to first try running getScmStatusV2() with fallback option. This does not retry `hg status` while a checkout is in progress.
Reviewed By: simpkins
Differential Revision: D18209899
fbshipit-source-id: e7a77b902f5a0ee624e4ea3185a1901bdac090e6
Summary:
Now that we've transitioned to the newer redirections
configuration we can remove the older bind mounts configuration
parsing, and that is what this diff does.
This is helpful because in situations where the user has run
`hg co null` as part of some troubleshooting advice, the legacy
bind mounts would remain mounted and obstruct some of the
steps that would have fixed up the users repo.
Reviewed By: pkaush
Differential Revision: D18337246
fbshipit-source-id: 23f27787d609e1c38a9c98b8b6596bb40743b9ca
Summary:
The purpose of this command is to unmount/unlink any configured
redirections without removing their configuration.
The intent is to call this for a mount when we are unmounting; I'll do
that bit in a follow on diff.
Reviewed By: pkaush
Differential Revision: D18801872
fbshipit-source-id: 096d9595091da72aa85f4259cbab022a1fe0c01f
Summary: Even if blobs have different hashes, they could have the same contents. For example, if between the two revisions being compared, if a file was changed and then later reverted. In that case, the contents would be the same but the blobs would have different hashes. Currently, `getScmStatusBetweenRevisions()` would report false positives in this case. This is also needed so we do not report false positives in `getScmStatus()` when hit this code path
Reviewed By: simpkins
Differential Revision: D18647086
fbshipit-source-id: 66e12648a24fd7e5612eee5e599a5b81c7c5f2d1
Summary: This reads `enforceParents` from a config instead of always assuming true for `getScmStatusV2()`. This will allow a easy kill switch in case throwing errors from this thrift call causes issues with something that calls hg status
Reviewed By: simpkins
Differential Revision: D18258164
fbshipit-source-id: 1ae421a941c01a678d25d5453c771262b03558d0
Summary: make the error message returned in the case of out of date parents during a new status call more user friendly and provide possible remediation instructions
Reviewed By: simpkins
Differential Revision: D18328835
fbshipit-source-id: b214f45bb055d008db8b233ddd2a1843332db838
Summary:
Merge the fb-mercurial code into the Eden repository, under the
`eden/scm` subdirectory.
Reviewed By: quark-zju
Differential Revision: D18445774
fbshipit-source-id: fc3307f9937e0c7e1c8f7d03c5102c4fe5dedb10
Summary:
Add a new thrift API for computing the difference between the working
directory and a given source control commit.
This has the following differences from the old getScmStatus() commit:
- The parameters are accepted in a GetScmStatusParams structure now.
This makes it easier for the server-side C++ implementation to tell which
parameters have actually been specified by the caller. This will make it
easier to extend this API in the future without having to replace it with a
new function call again.
- The return value is a GetScmStatusResult, which includes both the ScmStatus
and the EdenFS version number. This will allow code like `hg status` to get
both the status results and the EdenFS version in a single call, without
needing to make multiple separate thrift calls.
- This new call will return an error if the caller requests the status against
a commit that disagrees with EdenFS's view of the current commit. Because
the individual `hg` command line processes do not perform any
synchronization of their own when reading the working directory parent,
they can often call EdenFS with stale parent information, or while a
checkout is currently in progress. This new behavior will reject the
request with an error, rather than having EdenFS perform a potentially very
expensive status computation when the results probably aren't actually
useful to the caller anyway.
Reviewed By: chadaustin
Differential Revision: D15110218
fbshipit-source-id: ebc2f74dafc090d4fd245de8e4d62e2b086500dd
Summary: Adds a non-optional EdenErrorType struct for EdenError, this can be used in case of special error case handling of errors without error message parsing. Currently this is just passed along and not consumed anywhere in the client, but later in the stack is used for specific retry of checkout if "CHECKOUT_IN_PROGRESS" on the consuming side.
Reviewed By: chadaustin
Differential Revision: D18139917
fbshipit-source-id: b3f2ec4c480fc5246ff2f46d09c436021bad8b61
Summary: Fastmanifest is going away, remove it from the test.
Reviewed By: chadaustin
Differential Revision: D18145000
fbshipit-source-id: ee75ebe4eda19caca92fd0a84bf0ae9f48112167
Summary: Formatting had diverged in a few places. Fix that up.
Reviewed By: fanzeyi
Differential Revision: D18123219
fbshipit-source-id: 832cdd70789642f665a029196998928a9173be81
Summary:
A recent change removed the revision number from the rebase output, fix the
EdenFS test to not have it too.
Reviewed By: genevievehelsel
Differential Revision: D17954310
fbshipit-source-id: 6c1db48086af4b7b138e6c3f4ef0bb362d2256f8
Summary:
Instead of having accessors for every config setting in EdenConfig,
just expose the ConfigSettings directly.
Reviewed By: fanzeyi
Differential Revision: D17847805
fbshipit-source-id: 8c6c1010c010113cf859677449797ea916f2a2a5
Summary:
Some Unix applications (notably, nfsd) create regular files using vfs_create, which ends up invoking the `mknod` system call rather than `open`, which for historical reasons only supported socket creation with Eden. However, since Eden supports regular files, we can broaden the FUSE mknod handler to support regular files as well.
For context, see https://github.com/GoogleCloudPlatform/gcsfuse/issues/137#issuecomment-155273363
Reviewed By: chadaustin
Differential Revision: D17792424
fbshipit-source-id: 466fcbcb3bcb587e731bc8b2a3e0f1508ff1f4e4
Summary: D17766371 added an `update --merge` state. Teach eden tests about it.
Reviewed By: wez
Differential Revision: D17837836
fbshipit-source-id: a95ed326bf435f7340d7910307c8c5c761812514
Summary:
## Backstory
Pyre was throwing errors in my diff (D17747558) regarding an extraneous fixme. Turns out PyreBot has been adding and removing these fixmes during version updates (see D17135557 and D16183608), so I suspect it's something to do with the Pyre version. Anyways, I figured it'd be easier to do the annotation than to remove the fixme and risk Pyre throwing the same error in a later diff.
## What I did
I added the Key type (Path) and Value type(ExpectedFileBase). mypy then started throwing an error regarding __iter__ returning the wrong type because it wanted an iterator over keys rather than values. Fixed that and added .values() to the for loop.
Reviewed By: genevievehelsel
Differential Revision: D17806135
fbshipit-source-id: c07feee33db78a9bff19ba9856a7047657b8c63e
Summary:
Update the CMakeLists.txt files to support building the hg integration tests.
At the moment this only includes one of the test files (`status_test.py`).
I have not verified if tests from the other modules pass yet or if they need
any additional tweaks to work in CMake-based builds.
Reviewed By: pkaush, fanzeyi
Differential Revision: D17678991
fbshipit-source-id: 4a5ee5a8d6039d9d2a635c7027897bbeed14f8c0
Summary:
Add initial support for building and running some of the integration tests
with CMake. For now this just runs the tests from basic_test.py, just to
confirm that most of the framework code works in CMake-based builds.
Many of the other tests should also work as well, but a few of them we may
want to disable for CMake-based builds. e.g., a couple of the tests depend on
hypothesis, and we would need to include hypothesis as a dependency. Some of
the tests that use systemd might also require a little more work to get
working.
Reviewed By: fanzeyi
Differential Revision: D17659026
fbshipit-source-id: 67420fda9e1021a0cddee2d385fd21e34fb2fd70
Summary:
Force a reference to the edenfsctlPath flag, otherwise the
linker will discard it and a large number of tests will fail.
Reviewed By: simpkins
Differential Revision: D17683222
fbshipit-source-id: b7cb29e74af85b544f45a228770ad2613c8e6efc
Summary:
This diff removes the logic that consumes the legacy bind
mount list and mounts them on startup. That functionality has been
replaced with the eden redirect command.
Instead of performing the bind mounts in the server, the server will
now run `eden redirect fixup` to apply that configuration.
This diff also changes the behavior of performBindMounts: previously, if the
bind mount setup failed, we would tear down the entire repo mount. Since we're
now spawning an external process, it is much more likely that something might
fail and result in a bad experience, so we no longer bail out in that case:
we'll continue and leave the bind mounts as-is. The user can then use `eden
doctor` or `eden redirect fixup` to sort things out.
Reviewed By: simpkins
Differential Revision: D17236366
fbshipit-source-id: 8b004551a076216f0e5448942f00b5195ee18803
Summary:
Change the `//scm/hg:hg` target to use an `sh_binary()` rule that invokes the
`:hg_rust` binary with the proper environment so it can find its dependencies,
rather than copying the binary and all of its dependencies into a new
subdirectory.
In dev mode builds the `hg_rust` binary isn't guaranteed to work anywhere
other than its original location, due to the way that dev mode builds use
`$ORIGIN` in the binary's `RPATH` setting. This happened to work up until now
as the hg_rust binary did not have any separate libraries, but I plan to add
one on the `chg` library.
Reviewed By: quark-zju
Differential Revision: D17109104
fbshipit-source-id: ae8bb1126969f012d1d2fb7d04e80867a310b9a8
Summary:
Add a flag to cause `eden start` to exit successfully without doing anything
if EdenFS is already running. This flag makes it slightly easier for
automation to ensure that EdenFS is running, without logging warnings if
EdenFS was already running.
I also cleaned up the error message slightly when `eden start` is used
without this flag and fails if EdenFS was already running. Previously the
exception thrown was unhandled so it also printed a python backtrace. Now the
code throws an exception that is caught by the higher level command line code,
so it is printed in a more user-friendly way.
Reviewed By: wez
Differential Revision: D17440486
fbshipit-source-id: d7661ef7be7159bf5542b20e99a0b5495690e5a2
Summary: This makes it a bit more human friendly
Reviewed By: chadaustin
Differential Revision: D17249465
fbshipit-source-id: 40d5afc77ded34237e1860d5b91e9257a732e480
Summary:
D17236366 will disable the getBindsMount thrift call and
remove the internal source of data about bind mounts. We instead
have a more current set of data from the `redirect` command, so
tech `eden chown` to use that data.
Reviewed By: chadaustin
Differential Revision: D17249433
fbshipit-source-id: 853f24e729814c501768e9834765e1be283d6aac
Summary:
Integration test helpers relied on an implicit gflags include. Make
that explicit so they compile against open source gflags and glog.
Reviewed By: wez
Differential Revision: D17264335
fbshipit-source-id: e336423b71c0f15e29b0e4ad604328b7624080a8
Summary:
Make sure the contents of the special `.eden/` subdirectory are correct each
time we mount a checkout. Before we would generally only set up the contents
of this directory if it didn't previously exist.
Now the code verifies that the contents of each symlink in this directory are
correct, and recreates the symlink if needed.
This allows EdenFS to automatically repair the contents of this directory even
if the checkout or its `clients` directory has been manually moved.
Reviewed By: wez
Differential Revision: D17279413
fbshipit-source-id: e24e7530f44fff94ebb6f67174aaf78c9b498d6b
Summary:
Update the fsck code to save any orphaned symlink inodes that it finds as
symlinks in the repair archive directory, rather than saving the contents as a
regular file.
Reviewed By: wez
Differential Revision: D17170346
fbshipit-source-id: 4cba8b27233b728114a80a327ab519b039297aea
Summary:
Use the new `OverlayChecker` class to automatically scan for errors and
attempt to repair them if the overlay was not shutdown cleanly the previous
time it was used.
Reviewed By: wez
Differential Revision: D16596601
fbshipit-source-id: 9923565b101ba953e92909e502be6ef5895c5cbd
Summary:
This was causing problems on macos where various tools
would enumerate and helpfully try to preserve attributes across
copies. On macos this would result in appledouble metadata files
being created to track the metadata in the destination file,
which clutters up the repo and has surprising secondary effects
such as being picked up by glob operations in cmake build rules.
This diff simply stops enumerating the extended attribute.
Reviewed By: fanzeyi
Differential Revision: D17140414
fbshipit-source-id: 2924657dc75b900baf70595edfa72e5d0521a697
Summary:
The snapshot generation code and many integration tests create
repositories. By default, they were creating flatmanifest
repositories, which are on their way out. Instead, create tree-only
repos.
Reviewed By: strager
Differential Revision: D17066151
fbshipit-source-id: f99a9543440da6fd7cce0065c3cd7f91a59a02d5
