Summary:
`eden du --clean` currently fails with
```
Fatal Python error: initfsencoding: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'
```
Full error: P149352812
It looks like this is because Buck expects to run with a different python, so
here I clear out the PYTHONHOME variable before we start buck.
This reuses out logic used elsewhere to clean up the environment before
calling buck.
Reviewed By: wez
Differential Revision: D24904105
fbshipit-source-id: 73587c52aff3ea00f68339eb44e7042329de2e44
Summary:
Add a TraceBus to HgQueuedBackingStore and allow tracing import events over Thrift.
This powers a new `eden trace hg` command that allows a live view of
tree and blob imports.
Reviewed By: genevievehelsel
Differential Revision: D24732059
fbshipit-source-id: 525152fe39047160a68c1706217a06a00a6dbef1
Summary:
This is what `eden debug` shows:
modified (m, a, t, e, r, i, a, l, i, z, e, d)
Enumerate all potentially-modified inode paths
Reviewed By: chadaustin
Differential Revision: D24908570
fbshipit-source-id: 62e91b6f0212c70de4bb1705539d3674e6e000d9
Summary:
The rest of the non debug commands use - instead of _.
Lets flip the prefretch profiles cli to be consistant.
Reviewed By: genevievehelsel
Differential Revision: D24910172
fbshipit-source-id: a5f18a9c9d5fb4ef9417f14ef9d053cdc1599d76
Summary:
The second line of the summary got cut off due to a bad multiline comment
signals_oops
Reviewed By: wez
Differential Revision: D24909886
fbshipit-source-id: 844778e7d47a2b7b413fdd0c4fa0ef71dec9dadb
Summary:
`eden prefetch` can trigger a bunch of wasted network requests to fetch metadata
when we are fetching files anyways. (These network requests are wasted since we
fetch the file contents and most of them are being throttled on sandcastle anyways.)
So lets skip metadata prefetching during eden prefetches.
Reviewed By: genevievehelsel
Differential Revision: D24658066
fbshipit-source-id: f8a32807a4e238222158f100cdd5ffa1b92fd833
Summary:
We will start opting-in and rolling prefetch profiles mvp out to users soon.
This is a switch to allow users to opt-in, us to gradually rollout, and to
quickly turn prefetch profiles off if this causes issues for users.
Reviewed By: genevievehelsel
Differential Revision: D24803728
fbshipit-source-id: 0456f2a733958b495e5d84f7177c99f3ef481f57
Summary:
There were `eden top` issues on MacOS that I thought had been fixed a while ago,
but it doesn't look like we caught them all. This should catch the remaining bug
in `eden top`.
Reviewed By: genevievehelsel
Differential Revision: D23743199
fbshipit-source-id: ca66748c7a8a26062caf934c8f2c1fd13d9ae69e
Summary:
Add a very simple debug command that simply prints all materialized
inodes under the given path. It is a quick way to uncover unexpected
writes (or unexpected failed dematerializations after checkou).
Reviewed By: xavierd
Differential Revision: D24378759
fbshipit-source-id: dc393d65506c74dbc0779732cdefd915cbbf9948
Summary:
Replace the old implementation of debugInodeStatus with the more
general traverseObservedInodes functionality, and add the ability to
customize its results with flags.
Reviewed By: xavierd
Differential Revision: D24300122
fbshipit-source-id: 0fbd3aa02575faa515fd7852441547d7de13426d
Summary: When I added `eden uptime`, I did not see that we already had `eden debug uptime`! Instead of having two, lets just use the one and remove it from debug. There seems to be a regression of the uptime I had implemented, so thats why I opted to use the implementation of the debug version.
Reviewed By: kmancini
Differential Revision: D24566844
fbshipit-source-id: d948a5303d475a543f51abbaea59f9c481dfeca2
Summary:
One of the unforseen effect of D24393690 (c5d631fd09) is that EdenFS was tied to the console
that ran `edenfsctl start`. As a result, killing that console would also kill
EdenFS, which is a bit unexpected.
Console in Windows are a bit complicated, and in theory, EdenFS could simply
call `FreeConsole` once it is started, but the side effect of this is that any
process started by EdenFS (say Mercurial), would create its own console as its
parent doesn't have one, specifying CREATE_NO_WINDOW when creating these would
lead to not having any output in the edenfs logs from Mercurial (D21820195 (223846d313))...
We could solve this if we could tell Windows to allocate a hidden console
window, but that API unfortunately doesn't exist.
So, the best we can do for now is to simply start EdenFS with the
CREATE_NO_WINDOW flag, this fixes the above issue, allows for the original
console to be killed without affecting EdenFS, but it brings another issue:
`eden start` no longer shows what is going on...
For now, this is the least worst solution, so we'll go with that. In the
future, we can imagine fixing EdenFS startup code to send its output to a pipe
that edenfsctl would read and print to its standard output.
Reviewed By: chadaustin
Differential Revision: D24607540
fbshipit-source-id: c18590052a96b7dd2938b589e92e808f38b5ef59
Summary:
We want to be able to prefetch profiles on pull. That means we will need
to pick commits that will be good to prefetch for. This adds a flag --commits
that allows specifiying commits explicitly (mostly for testing), and a
flag `--predict-commits` which will pick good commits to prefetch for.
Reviewed By: genevievehelsel
Differential Revision: D23889882
fbshipit-source-id: 74e61c0c9d443ca396f326b0d547b9fc21a6364b
Summary:
Often a prefetch profile will be activated by a tool that uses that profile, so
we will by default prefetch a profile when activating it.
Reviewed By: genevievehelsel
Differential Revision: D23772446
fbshipit-source-id: 0b2e14cfa6ce079e8f04790ac2e0d196db5966ee
Summary:
Blocking checkout and pull to prefetch files could make these commands very
slow when large prefetches are triggered. To keep slow prefetches from causing
poor experiences we can background them by default.
Reviewed By: genevievehelsel
Differential Revision: D23831663
fbshipit-source-id: 4fba5ab894b927a6aa3359b516cff604ad524485
Summary: Introduces an `eden logs` command to read a large chunk (1M by default) file into a paste. This is also added to the the eden rage report to get more insight into systems in which we cannot log in to view the logs (like laptops).
Reviewed By: kmancini
Differential Revision: D24146812
fbshipit-source-id: 991f1595b974eb01f77e86559a8413b0b09a24a4
Summary:
Sometimes you only want to trace writes into the EdenFS checkout, so
add the ability to run `eden strace` with `--reads`, `--writes`, or
both in order to see only those events.
Reviewed By: wez
Differential Revision: D24468539
fbshipit-source-id: a1b3c730987cf86ce3d39952c6a5e9c5edaddfc2
Summary:
Now that Linux/macOS startup no longer uses fork it becomes trivial to share
the same code with Windows. This improves Windows startup in several different ways:
- `edenfsctl start` now displays the status of the start process in the console
- `edenfsctl start` no longer has an arbitrary timeout after which it reports
a timeout even though EdenFS is still starting up.
This also kills a bunch of Windows specific code that is no longer needed.
Reviewed By: fanzeyi
Differential Revision: D24393690
fbshipit-source-id: 28100aec96da81c92d5b592353edceed332e2364
Summary:
Add an `eden strace` command that subscribes to all FUSE events for
the specified mount (or current directory), and streams them to stdout
with response latency measurements and the requesting process.
If there are any pending requests at the time `eden strace` is run,
they are printed first.
Reviewed By: wez
Differential Revision: D24038978
fbshipit-source-id: 59a2112dbdb1708571d12b04bdccaf9eca725cf7
Summary:
Buck-out can contain some files that we wont have the ability to read on mac os
(like .Spotlight-V100 and .Trashes). This means `eden du` will error out on
reading them.
We handle FileNotFound but not permission errors, lets treat them the same.
Reviewed By: genevievehelsel
Differential Revision: D24396252
fbshipit-source-id: 31e2ecfeff1783ae9c3479f59335d95a0d9b6de7
Summary:
Include opcode name and process name in eden.FuseCall Thrift structs
so we can use eden.FuseCall in a later diff in the stack.
Reviewed By: kmancini
Differential Revision: D24036420
fbshipit-source-id: fc6d8f3d174b85e07fac299a6f86b2b2d24f301d
Summary:
Chef writes root-owned temporary files in /etc/eden/config.d. If
edenfsctl attempts to read those configs at this time, it will
fail. Just skip all hidden files.
Reviewed By: kmancini
Differential Revision: D24377953
fbshipit-source-id: 765b7f096b62e6749e186adc3e90bbc7814079c8
Summary: Add option to redirect the eden rage output to stderr
Reviewed By: chadaustin
Differential Revision: D24210031
fbshipit-source-id: 736af81de6a41e49c2459a1cd8124123d6928323
Summary:
There's a bug in Thrift-py3 streaming support, where interrupting
iterating over an async stream leaves Thrift objects in a broken
state. Futures get dropped (and warnings are printed to the console)
but the destructors hang. Don't even try to garbage collect the heap
in that case.
Reviewed By: genevievehelsel
Differential Revision: D24032229
fbshipit-source-id: 5f16667fe6cfd27de1b39cc2974028729e18b214
Summary:
Adds a subcommand to `eden debug prefetch_profile` to fetch all the files
for a list of profiles or all the active profiles for a checkout.
These profiles (files) can contain lists of file names, or globs of files, eden
will be able to resolve them since this uses the existing prefetch code.
I opted to put this command under `eden debug prefetch_profile` instead
of `eden prefetch` since the command may change to call into eden with out
the list of files (letting eden resolve the active profiles and names). Then
it will no longer resemble prefetch and so long term it may be better homed
here.
Reviewed By: genevievehelsel
Differential Revision: D23771483
fbshipit-source-id: 12af81d40cc495efd381e3c3a2df645d72053ccd
Summary: To be more Windows compatable, we should move away from using `sh -c`. We don't use shell=True because that is susceptible to shell injection. As long as we don't close stdin until we're done, using Popen raw should be fine.
Reviewed By: xavierd
Differential Revision: D24151425
fbshipit-source-id: c0bcc883af948491862e8ce0cee56bcbe98e04f1
Summary:
Formats a subset of opted-in Python files in fbsource.
Black formatting was applied first, which is guaranteed
safe as the AST will not have changed during formatting.
Pyfmt was then run, which also includes import sorting.
The changes from isort were manually reviewed, and
some potentially dangerous changes were reverted,
and the directive was added to those
files. A final run of pyfmt shows no more changes to
be applied.
Reviewed By: zertosh
Differential Revision: D24101830
fbshipit-source-id: 0f2616873117a821dbc6cfb6d8e4f64f4420312b
Summary:
On Windows, the du command line application doesn't exist, thus we cannot use
it. Instead, we can simply re-implement the du functionality in Python and use
that on all platforms.
Reviewed By: chadaustin
Differential Revision: D24030269
fbshipit-source-id: e86c1bcdeac7eeca70201f6fde48c20ef7e305a6
Summary:
stats local-store hardcodes the column names and has diverged from the
source of truth in KeySpace.h. Bring it back up to date. Ideally,
there would be a Thrift call to retrieve the column information, but
genevievehelsel might end up replacing this stuff soon.
Reviewed By: genevievehelsel
Differential Revision: D24027548
fbshipit-source-id: 5e5195585025081969865f8d3651e742f721ea09
Summary:
Prefetch profile can now be used in automation to register profiles and to fetch
them. Since it will be used out side of the team it seems like this should be
moved out from under debug.
Reviewed By: genevievehelsel
Differential Revision: D23773337
fbshipit-source-id: b7f646ec202653233ed82528380d91f583a8341e
Summary:
This will allow us (and maybe users) to remove a users active profiles
easily when investigating issues. (And also will be useful for testing.)
Reviewed By: genevievehelsel
Differential Revision: D23517409
fbshipit-source-id: d57e595770b5a60726e2efe9cbf21c89dd694cc0
Summary:
This will allow tools to activate prefetch profiles relavant to their workflow.
Eventually this should also cause eden to prefetch the profile for the current
commit, but for now it just activates the profile.
Reviewed By: genevievehelsel
Differential Revision: D23517412
fbshipit-source-id: 172a844ca00bf3bcf14f4b94c25e8256e516b737
Summary:
This will allow us (and maybe users) to see a users active profiles
easily when investigating issues. (And also will be useful for testing.)
Reviewed By: genevievehelsel
Differential Revision: D23517410
fbshipit-source-id: 7a592f7b39c887600307a4f08668ca1ee9b982bf
Summary:
Assuming the format for active prefetch profiles as discussed here:
https://fb.quip.com/v8yFAzeGx8TU
This allows the cli to read in and write out active profiles, allowing the cli
to manipulate them in the following changes. This will allow tools to activate
profiles useful for that workflow. Additionally, it will allow us (and maybe
users) to see/remove a users active profiles easily when investigating issues.
Reviewed By: genevievehelsel
Differential Revision: D23517411
fbshipit-source-id: 8ab18a1aeccf8dcdd04aca826d61070fa0aedd33
Summary: This diff fixes the eden redirection tests so it runs on Windows.
Reviewed By: xavierd
Differential Revision: D22958766
fbshipit-source-id: 45d26587831ed74d6bd7912b22c7c955b077f571
Summary:
Rather than switch all of Eden at once to the thrift-py3 client,
rename get_thrift_client to get_thrift_client_legacy so uses of the
new client can be introduced piecemeal.
(I did try migrating everything at once but it's been quite painful.)
Reviewed By: fanzeyi
Differential Revision: D22423399
fbshipit-source-id: 9e6d938b90fff9fc3266ba20bc77e880e7f5b1aa
Summary:
There was a time when getRegexExportedValues was not in open source
fb303, but that time has long passed.
Reviewed By: kmancini
Differential Revision: D23745295
fbshipit-source-id: 4702068f0bb7350467e42439444b3f4d75aeec76
Summary:
ghostbooleans
Apparently I didn't test for the positive case in my previous diff that introduces this check :(
Reviewed By: xavierd
Differential Revision: D23698179
fbshipit-source-id: 95a28cc13bff5e325214b6a398e19c821b5ae17f
Summary: For ease of consumption, remove the descriptive line and the extra newline at the bottom of the generated prefetch profile. Also, sort the files for smaller generated diffs upon iteration.
Reviewed By: kmancini
Differential Revision: D23683153
fbshipit-source-id: e2bd510d5fbd7095f199e70b2556b84e0984a914
Summary:
We intend to rename the edenfs device type to include a colon (and
possibly the backing repo basename). In preparation, update code that
detects edenfs mounts to include anything that starts with "edenfs:".
Reviewed By: genevievehelsel
Differential Revision: D23520008
fbshipit-source-id: 280f7617d5c96e23d548041b3482bca388076a7b
Summary:
For some unknown reason, we weren't setting this on Windows, which meant that
whenever edenfs would need to call edenfsctl (like at mount time when fixing up
redirections), it would always use the binary found in the PATH. While in most
cases this is OK, this is not the intended behavior for tests that are expected
to use the just compiled binary, not the one in the PATH.
Reviewed By: genevievehelsel
Differential Revision: D23653027
fbshipit-source-id: f1cc977e44b10c379d2b90bc7972bfec1fccad23
Summary:
The AccessCount fields were recently renamed to start with fsChannel instead of
fuse to be more platform independant. This broke edenfsctl due to it using the
old names.
Reviewed By: wez
Differential Revision: D23633574
fbshipit-source-id: 2a5fc73c47d2f0a6db407ecfeaf85992b7932c10
Summary:
This diff fixes a bug on Windows when the redirection target is a non-empty directory. As seen in P141872812
This doesn't make the exception go away but generate a more meaningful error message so the user can act on it.
Reviewed By: xavierd
Differential Revision: D23605233
fbshipit-source-id: 2d2bde0e9cd94323a6537ebcec29a4c15868806d
