Summary: 90% of the time, it's more useful to see the end of the log than the beginning.
Reviewed By: simpkins
Differential Revision: D19560473
fbshipit-source-id: f9b5a922d284053d6afc8ea889586826e3c58582
Summary:
Sadly, we didn't derive as much value from these as we would
have liked, and now they are a source of flakeyness in our tests.
Reviewed By: genevievehelsel
Differential Revision: D19460792
fbshipit-source-id: 48c82cc2d1fdbd81bece4057e93799fbcc4f4725
Summary: Passing the checkout path was throwing an error on Windows - "error: argument of type 'WindowsPath' is not iterable". Passing it as string to fix it.
Reviewed By: simpkins
Differential Revision: D19507974
fbshipit-source-id: 2a91dc37d2497a30a70c684f5298410956a6acca
Summary: `hg gc` can now clean things up there, so suggest that.
Reviewed By: quark-zju, fugalh
Differential Revision: D19413816
fbshipit-source-id: 1c6c08ed9fb4757390883d908531d9b3a7da302d
Summary: adds a cli debug command to inspect the working copy parent. by default just returns eden's snapshot contents, but adds optional --hg flag to print Mercurial's dirstate information
Reviewed By: chadaustin
Differential Revision: D19167518
fbshipit-source-id: b65e112df6abe4e0e7a8a528a90b2e3d17297e66
Summary:
The doctor tests for available disk space checked the host's actual
disk, causing them to fail when run on machines with full disks. Add a
fake and use it by default.
Reviewed By: simpkins
Differential Revision: D19145853
fbshipit-source-id: b4202a4eabffd8b7a968de9fb4370c0a0ca99d4d
Summary: There is one instace of `getScmStatusBetweenRevisions` in use - it is used in the eden cli in a hacky way to check if a commit hash is valid. Since this is not used anywhere else in a meaningful way, this replaces that use case with a hg call and depreciates `getScmStatusBetweenRevisions`
Reviewed By: simpkins
Differential Revision: D18690026
fbshipit-source-id: 02bd2c20a0f631ec41116f9fd4e18d14369298ef
Summary:
This fixes a regression in `eden top` introduced this summer where it
would gradually add to the terminal's scrollback. The cause was
accidentally using `window.clear` instead of `window.erase`.
Reviewed By: genevievehelsel
Differential Revision: D19061316
fbshipit-source-id: 4ef4c7bcd00e30ff5d3b80406ea01a476346faf9
Summary:
Use all available lines, now that we ignore the spurious errors that
can be thrown when calling `addnstr()` on the last line.
Reviewed By: wez
Differential Revision: D18826918
fbshipit-source-id: 7e5739207ba1a5bfa0d290da1345c9d9606b89fb
Summary:
Add a wrapper function around `addnstr()` that silently ignores any errors.
When writing to the last cell of a terminal curses will successfully display
the data, but report an error since the logical cursor cannot be moved to the
next cell. This error is not important and can be ignored.
The errors reported by the `curses` module unfortunately are not very useful
(they normally just report `addnwstr() returned ERR`). For now we silently
ignore all errors, although this diff does contain some stub code that can be
enabled to examine them during development.
Reviewed By: wez
Differential Revision: D18826917
fbshipit-source-id: a4c5394b49aea1334def1d047eec7b5e743fb9f2
Summary:
Now that we've transitioned to the newer redirections
configuration we can remove the older bind mounts configuration
parsing, and that is what this diff does.
This is helpful because in situations where the user has run
`hg co null` as part of some troubleshooting advice, the legacy
bind mounts would remain mounted and obstruct some of the
steps that would have fixed up the users repo.
Reviewed By: pkaush
Differential Revision: D18337246
fbshipit-source-id: 23f27787d609e1c38a9c98b8b6596bb40743b9ca
Summary:
The intent is to allow macOS to make a best effort attempt
to detach any disk images associated with redirections.
D18795800 taught redirect to detach rather than a simple unmount,
and we need to give it an opportunity to detach when we unmount
gracefully so that all resources are released on shutdown.
The implementation of this diff ties into the same mechanism that
we use to shutdown the buck daemon when we unmount or restart which
has the nice side effect of showing the output from the underlying
`eden redirect unmount` command.
In an earlier iteration of this diff I tried making the server code
run `eden redirect unmount` but it happened too late in the process
to be effective: the thrift server had already been torn down in the
shutdown case and we need privs to perform the unmount.
Reviewed By: pkaush
Differential Revision: D18804080
fbshipit-source-id: 0b409130752121c56a46c9b2e46b50e5abee8200
Summary:
The purpose of this command is to unmount/unlink any configured
redirections without removing their configuration.
The intent is to call this for a mount when we are unmounting; I'll do
that bit in a follow on diff.
Reviewed By: pkaush
Differential Revision: D18801872
fbshipit-source-id: 096d9595091da72aa85f4259cbab022a1fe0c01f
Summary:
unmount will unmount from the VFS but still leave the image
associated with the disk image machinery and visible to `hdiutil info`.
This leaves resources associated with the mount even after it has
logically gone away, and that is undesirable.
This diff switches to using `detach` instead which unmounts and removes
the image from the disk image state.
The effect of this is that running `hg co null ; eden redirect fixup`
will completely unmount the redirections.
However, it doesn't cause them to be unmounted when the eden daemon is
stopped: that is something to address in a separate diff.
Also worth noting: only HFS+ mounts get detached successfully using this method. APFS mounts will remain regardless, so its another reason for switching to HFS+ in D18795799.
Reviewed By: pkaush
Differential Revision: D18795800
fbshipit-source-id: dfc86d86016a0c78e67f6ae2651db681669a5b14
Summary:
Update Eden's CMake build to use the new `install_fb_python_executable()`
function to install `edenfsctl`
Reviewed By: wez
Differential Revision: D18774538
fbshipit-source-id: 462c0127d79edcb6235a629cb97ea481493e1906
Summary:
We've seen occasional weirdness with "resource temporarily unavailable" when working with disk images.
The internet, in the form of this stackoverflow post: https://stackoverflow.com/q/48368389/149111
suggests that this is related to APFS.
This commit switches over to HFS+ in the hope that we won't see it any more.
Reviewed By: chadaustin
Differential Revision: D18795799
fbshipit-source-id: 68e08e852770e311bcf04a8d12cb20670babf889
Summary: make the error message returned in the case of out of date parents during a new status call more user friendly and provide possible remediation instructions
Reviewed By: simpkins
Differential Revision: D18328835
fbshipit-source-id: b214f45bb055d008db8b233ddd2a1843332db838
Summary:
To avoid scenarios where hard reboots prevent the SNAPSHOT page from
getting flushed, atomically write the file upon clone.
Reviewed By: simpkins
Differential Revision: D18382583
fbshipit-source-id: 1c4de46165587635f71744daf4d2a8c71c58cc5f
Summary:
We've seen some weird situations where the VFS has some surprising
layering. My favorite example so far had 4 layers of mounts, with the lower 3
layers being dead edenfs mounts that were still present in the mount table but
that had no associated fuse server process.
In that situation the mount table looked approximately like this:
```
/some/path/repo <-- top of vfs
/some/path/repo/buck-out invisible because of the mount above
/some/path/repo an older instance
/some/path/repo/buck-out invisible because of the mount above
/some/path/repo an older instance
/some/path/repo/buck-out invisible because of the mount above
/some/path/repo the oldest instance
```
aside from being weird this shouldn't impact our behavior, except that it was
because our logic to uncover the effective bind mount configuration just looks
at the mount table and trusts that one of the matches mounts is visible to the
VFS.
The problem is that the top layer has no buck-out bind mount redirection which
leads to buck writing lots of data to the overlay.
To resolve this we need to verify that the mount table entry is visible
in the VFS and for that we can use our existing `is_bind_mount` helper
function.
Reviewed By: pkaush
Differential Revision: D18335468
fbshipit-source-id: 509c8b43f5e69f0a42c292a7011271d81ab97919
Summary:
Fix the `getStatInfo()` code to avoid walking all loaded inodes. That
behavior was okay early on when we were first developing EdenFS and this
API wasn't used in many places. However, `eden rage` and `eden stats` both
exercise this code path, and it seems pretty bad to walk all loaded inodes in
order to return some stats.
I removed the counts about number of materialized inodes for now, since I
don't think we're really using this metric much.
Reviewed By: chadaustin
Differential Revision: D17578306
fbshipit-source-id: 55ab0209745869b160e91167d6cff7d95f39a95a
Summary:
In order to optimize jobs using eden, we want to be able to see which directories we write/read a lot of files to/from. This information allows us to see if we can improve performance by adding certain directories to prefetch, or by symlinking directories that we write to often.
This diff adds to the summary a list of directories with at least 1000 files written to/read from them.
Reviewed By: fanzeyi
Differential Revision: D17747558
fbshipit-source-id: cf59beeee7c2eda9dea0dfa99f7ab92be9b33e1e
Summary:
Since it has become cheap to fetch file sizes, we made a change in D17633671 to ignore the --sizes flag and always fetch sizes.
Now that we have removed the use of --sizes flag on the Sandcastle side in D17729705, we can remove the flag from the cli as well.
Reviewed By: fanzeyi
Differential Revision: D17746946
fbshipit-source-id: 1a943e32db6a6f6754a20d8e8fa3ac25f241f778
Summary:
One of our diagnostics attempts to run `nuclide-connections`,
and prints a warning if that tool is not found.
In our macOS environment that tool is never present so it always emits
a warning that is not actionable.
This diff suppresses that particular warning.
Reviewed By: fanzeyi
Differential Revision: D17745753
fbshipit-source-id: 3394e88d443383880b632fca04a1ec074e46a9bd
Summary:
The current debug file_stat tool returns a list of read and written files with their corresponding sizes. This is difficult to read and understand.
We add a summary section with:
- file count
- total bytes read/written
- total large files (> 10MB) read/written
- list of large files
The existing list of read/written files are moved into a `details` section.
Reviewed By: fanzeyi
Differential Revision: D17633963
fbshipit-source-id: a0b14d900e13590214b64fd9bbd7db2a0c69b300
Summary:
D16836053 -> Diff where --sizes arg was added. Back then, sizes were computed locally and added significant overhead, so it made sense to put this feature behind a flag.
D17080186 -> Diff where function was updated to fetch sizes from thrift call instead of computing it locally
Eden CLI seems to be deployed on Sandcastle on a weekly cadence, so I'm holding off on removing the --sizes flag for now. We can wait till this change is deployed, then make the change to the callsite [here](https://our.intern.facebook.com/intern/diffusion/WWW/browse/master/flib/intern/sandcastle/util/SandcastleEdenUtil.php?commit=1001229135&lines=102) before coming back and removing the flag completely.
Reviewed By: fanzeyi
Differential Revision: D17633671
fbshipit-source-id: c640e5803d1c221f6912c3b4f995a2716c5e520b
Summary: The file_size attribute for an inode is not always defined because it can refer to a directory. However, since `LoadedNode`s are always regular files, we can enforce file_size to be always an `int`.
Reviewed By: chadaustin
Differential Revision: D17683020
fbshipit-source-id: b79275c2146e5304bdef02c3228cc0bf35456095
Summary:
The Python 3 zipapp module always exits with return code 0 if the main
function returns. Therefore we need to add our own wrapper function that
handles calling `sys.exit()` if we want to exit with a non-zero status.
Reviewed By: wez
Differential Revision: D17678021
fbshipit-source-id: 5944c75d8e765ab3e3fa038fa6f2eee95ab0e780
Summary:
The doctor code can't see into the newer redirection
config, which makes it only partially valid. In addition,
we run `eden redirect fixup` post-update; that supersedes
and replaces the doctor functionality.
I've also removed the associated tests: it's a shame that
they can't be re-purposed to work with the redirection code,
but their model is too awkward to use in that way--those tests
fake up aspects of the mount table and mount commands in layers
that are difficult to holistically fake up.
We have integration tests for the redirect functionality, so
we're not leaving things totally uncovered.
Reviewed By: simpkins
Differential Revision: D17252083
fbshipit-source-id: 8a576fb457bf0a4082bde1770085e809484ee73a
Summary:
This diff removes the logic that consumes the legacy bind
mount list and mounts them on startup. That functionality has been
replaced with the eden redirect command.
Instead of performing the bind mounts in the server, the server will
now run `eden redirect fixup` to apply that configuration.
This diff also changes the behavior of performBindMounts: previously, if the
bind mount setup failed, we would tear down the entire repo mount. Since we're
now spawning an external process, it is much more likely that something might
fail and result in a bad experience, so we no longer bail out in that case:
we'll continue and leave the bind mounts as-is. The user can then use `eden
doctor` or `eden redirect fixup` to sort things out.
Reviewed By: simpkins
Differential Revision: D17236366
fbshipit-source-id: 8b004551a076216f0e5448942f00b5195ee18803
Summary:
mostafaeweda and otawfik observed that repeating the same `eden redirect
add` call with the same parameters was tearing down and re-mounting the bind
mount. That shouldn't matter but it isn't as fast as not doing anything, and
sometimes hdiutil on mac can get into a weird state.
This commit compares the existing configuration against the new redirection
entry that we intend to add before applying the config; if they match then
it prints a note about not doing anything and then returns.
Reviewed By: mostafaeweda
Differential Revision: D17639592
fbshipit-source-id: 48d70d11e14a1a191ae655ad7a990c1c90c14953
Summary: In the case of a hard reboot, ensure the per-checkout config.toml isn't corrupted.
Reviewed By: wez
Differential Revision: D17672198
fbshipit-source-id: 98141b411e70ed8b30052f989be8161c3ffcc354
Summary:
hdiutil is horribly verbose, which in turn causes `eden redirect fixup`
to be horribly verbose on macos, so let's supress its output when successful.
Reviewed By: simpkins
Differential Revision: D17254318
fbshipit-source-id: ed2508e554d70bdcde0f0546e24dd6155e05d258
Summary:
The sparse images that we mount in place of bind mounts on macos
can grow over time, and don't automatically shrink if their contents are
purged.
This commit teaches `eden gc` to run `hdiutil compact` on these images
to reclaim space.
The output from `hdiutil compact` is really verbose and noisy, so we
suppress it unless there is an error.
Reviewed By: simpkins
Differential Revision: D17254235
fbshipit-source-id: 944b3824717c8b46d8170ec729c04a93323ab7f0
Summary:
We're seeing an issue where for an Eden instance started
as root a subsequent chown doesn't chown the root of the bind mount
and thus makes it impossible to write into the directory.
Reviewed By: simpkins
Differential Revision: D17620135
fbshipit-source-id: 4ee511234c551bbfc7330a60ee85fc5efd329d73
Summary:
It's convenient to manipulate Thrift output with jq sometimes, so add
an option to try to encode the resulting Thrift object as JSON.
Reviewed By: simpkins
Differential Revision: D17468559
fbshipit-source-id: b1f4639b6ea0d366400c8283cd7095bb4e0728de
Summary:
Use fdatasync (or fsync on macOS and Windows) and os.replace to write
Eden's config.json.
Reviewed By: wez
Differential Revision: D17465682
fbshipit-source-id: 0e92a58653befc84d45eb815bc0a02b8c2df3d39
Summary:
Update the getdeps manifest and Eden's CMake files to indicate that the
edenfsctl program depends on python-toml.
Reviewed By: chadaustin
Differential Revision: D17401215
fbshipit-source-id: f512678d8bca9c7b2b4d25bf9c3ecd7eed825de9
Summary:
`require_checkout` returns an EdenCheckout, so infer_client_from_cwd
is no longer required.
Reviewed By: wez
Differential Revision: D17485008
fbshipit-source-id: b6ff6d4b28a48b56bd56fe7116313d6a64642f1e
Summary:
When passing a non-Eden path to `eden info`, it would spit out a
Python stack trace. Fix that by using the standard checkout lookup
code.
Reviewed By: wez
Differential Revision: D17484897
fbshipit-source-id: beae64c4aa81af201e7560b44f0d103196e4ab35
Summary:
Add a flag to cause `eden start` to exit successfully without doing anything
if EdenFS is already running. This flag makes it slightly easier for
automation to ensure that EdenFS is running, without logging warnings if
EdenFS was already running.
I also cleaned up the error message slightly when `eden start` is used
without this flag and fails if EdenFS was already running. Previously the
exception thrown was unhandled so it also printed a python backtrace. Now the
code throws an exception that is caught by the higher level command line code,
so it is printed in a more user-friendly way.
Reviewed By: wez
Differential Revision: D17440486
fbshipit-source-id: d7661ef7be7159bf5542b20e99a0b5495690e5a2
Summary:
It's possible that status includes ignored files from
the overlay that are no longer accessible because a bind mount
is overlaid on top of the overlay!
Ideally we'd account for the space occupied by those files,
but for now we'd rather not error out when attempting to stat
them, so silently ignore that class of error.
Reviewed By: chadaustin
Differential Revision: D17415965
fbshipit-source-id: 96d467ee4e85ad5bdc730e4cfffe4c15a83a12ed
Summary: This makes it a bit more human friendly
Reviewed By: chadaustin
Differential Revision: D17249465
fbshipit-source-id: 40d5afc77ded34237e1860d5b91e9257a732e480
Summary:
Some of our internal repos have important post-update
hooks and we currently are not calling them for the initial clone,
so do that.
Reviewed By: chadaustin
Differential Revision: D17244812
fbshipit-source-id: 3424458710a08bf3bcd3331a09a643a90a6a63be
Summary:
D17236366 will disable the getBindsMount thrift call and
remove the internal source of data about bind mounts. We instead
have a more current set of data from the `redirect` command, so
tech `eden chown` to use that data.
Reviewed By: chadaustin
Differential Revision: D17249433
fbshipit-source-id: 853f24e729814c501768e9834765e1be283d6aac
