Summary: This should fix the test failure introduced in D28203778 (9bee308885) where fsck tests were not skipped on Windows.
Reviewed By: genevievehelsel
Differential Revision: D28227230
fbshipit-source-id: 97ef5717fbda567b48f02a60e1906e374fdc6766
Summary:
This applies the formatting changes from black v21.4b2 to all covered
projects in fbsource. Most changes are to single line docstrings, as black
will now remove leading and trailing whitespace to match PEP8. Any other
formatting changes are likely due to files that landed without formatting,
or files that previously triggered errors in black.
Any changes to code should be AST identical. Any test failures are likely
due to bad tests, or testing against the output of pyfmt.
Reviewed By: thatch
Differential Revision: D28204910
fbshipit-source-id: 804725bcd14f763e90c5ddff1d0418117c15809a
Summary: This just makes it more obvious _where_ `--force` should be passed.
Reviewed By: genevievehelsel
Differential Revision: D28119590
fbshipit-source-id: 1fbdb4428e9b89e7b66c959f874067485a91d534
Summary:
Currently we have limited test coverage of the NFS code. Let's start running
our integration tests on NFS mounts. We already duplicate tests to run them on
both Git and Hg repos using a python decorator. We can update this decorator to
run a copy of tests on an nfs mount.
This covers most of the tests, but a few tests do not use this decorator. See next
change.
Note some tests are currently broken, so I am using the same skip list functionality
we use for windows so we use a uniform framework.
Reviewed By: xavierd
Differential Revision: D27874662
fbshipit-source-id: c7d425830b691e395b5228d0e0f797f67987b4ec
Summary:
Some test runner don't properly redirect stdout/stderr of nested processes, or
even direct writes to filedescriptors. On these, debugging a test failure is
almost impossible for EdenFS as we rely on the test output to be interleaved
with the EdenFS logs to understand what the daemon is doing.
To solve this, we can simply create a thread that redirects the output of
EdenFS to sys.std{out,err}.
Reviewed By: kmancini
Differential Revision: D27570966
fbshipit-source-id: 6a8d5229d8d5d141e6ab423f7658744b42af46e3
Summary:
Currently eden on startup prints:
```
Starting edenfs (dev build), pid 190
Opening local RocksDB store...
Opened RocksDB store in 0.073 seconds.
Could not parse config.json file: couldn't read /var/twsvcscm/local/.eden/config.json: No such file or directory
Skipping remount step.
Started edenfs (pid 190)
Logs available at /var/twsvcscm/local/.eden/logs/edenfs.log
```
Would be convenient if it also printed session id, to be able to query scuba straight away.
Reviewed By: chadaustin
Differential Revision: D27522665
fbshipit-source-id: d7d4cf6c97bc551061761f2653375f208e393498
Summary:
I see in Sandcastle logs it was 10 minutes between `eden start` command and the following command.
So I'm curious, is it eden took 10 minutes to start or Sandcastle did something else but did not log it?
This little message in log will help to understand that.
Reviewed By: kmancini
Differential Revision: D27491709
fbshipit-source-id: 796c8db5abe49b056bd81b03ea57585a761c3cb6
Summary:
This is unused, and the Thrift file mentions that it should be fine to remove
in July 2020. It's now March 2021, time to kill it.
Reviewed By: chadaustin
Differential Revision: D26852134
fbshipit-source-id: 2872185edd834f889b78802210071d16b881e14c
Summary:
For now, this simply clone a repo with NFS, and nothing else, more of the
protocol needs implementing to support reading directories, files, etc.
Reviewed By: kmancini
Differential Revision: D26266144
fbshipit-source-id: e379e12126162f41d8d166bb53652e1e501de2e9
Summary:
Previously the code would result an exception raised while handling another
exception which is a little confusing. This diff fixes that.
Reviewed By: chadaustin
Differential Revision: D27100659
fbshipit-source-id: 8c6be4df62214c8e8d778478de66f271f7b84d3c
Summary:
As Rice is replacing the overlay, introduce some integration tests
around the subtler interactions of mode bits and status.
Reviewed By: genevievehelsel
Differential Revision: D26707622
fbshipit-source-id: 429ba404c1c0655f8fff393664bc5ba9e3b0301f
Summary:
It's silly to use `eden prefetch --no-prefetch` to efficiently glob
for filenames. Introduce an `eden glob` command which resolves a glob
relative to the current working directory.
Reviewed By: genevievehelsel
Differential Revision: D25450358
fbshipit-source-id: 45d6dc870d21510e51d5662c75e80385886899fc
Summary:
The world has moved on utf-8 as the default encoding for files and data, but
EdenFS still accepts non utf-8 filenames to be written to it. In fact, most of
the time when a non utf-8 file is written to the working copy, and even though
EdenFS handles it properly, Mercurial ends up freaking out and crash. In all of
these cases, non-utf8 files were not intentional, and thus refusing to create
them wouldn't be a loss of functionality.
Note that this diff makes the asumption that Mercurial's manifest only accept
utf8 path, and thus we only have to protect against files being created in the
working copy that aren't utf8.
The unfortunate part of this diff is that it makes importing trees a bit more
expensive as testing that a path is utf8 valid is not free.
Reviewed By: chadaustin
Differential Revision: D25442975
fbshipit-source-id: 89341a004272736a61639751da43c2e9c673d5b3
Summary:
Timeseries is memory intensive and not really required in the current context
it is being used.
Reviewed By: chadaustin
Differential Revision: D26315632
fbshipit-source-id: ee51c3ad8bef6fce152aa787c8c4602f0b499f92
Summary:
Previously, for configs that are only read once, EdenFS would have to be
stopped, the config written, and then EdenFS would be restarted. For Mercurial,
this increases the test time significantly as starting EdenFS takes ~20s.
Reviewed By: fanzeyi
Differential Revision: D26258174
fbshipit-source-id: a74d1e5be35044e95e5a7403f1bf28d557b613d2
Summary: This enables the repository to be mounted via NFS, and not FUSE.
Reviewed By: chadaustin
Differential Revision: D26229827
fbshipit-source-id: 5af5a47ebe5f1dd54df7707bf57d9b7476921f29
Summary:
The `assert_glob()` function in the glob tests previously only checked that
the expected number of results were returned. This updates the tests to
actually verify the returned paths. This also adds a few checks that verify
the behavior when the `wantDtype` parameter is True
Reviewed By: chadaustin
Differential Revision: D18903993
fbshipit-source-id: 5ed4c88160ffbda2e5fdcf089e210fda92868d56
Summary: **expected_sha1** was declared but not used in **xattr_test.py** file, removing this unused variable and variables which was used to creat **expected_sha1**
Reviewed By: xavierd
Differential Revision: D25899401
fbshipit-source-id: 26f0bb06d2c96e7f6754a4b821ffe4cf59a2f35e
Summary:
On Windows, since Mercurial doesn't yet build with Buck, we need to test
against the system Mercurial, thus remove the dependencies to //eden/scm:hg for
the tests. Also remove various dependencies that don't build yet on Windows.
This allows for the tests to run, but fail while trying to execute edenfsctl.par.
Reviewed By: kmancini
Differential Revision: D25807727
fbshipit-source-id: c2533eedc361cc6db9fdf2190476c3d52833139d
Summary:
Mercurial support files with `\` in their name, which can't be represented on
Windows due to `\` being the path separator. Currently, EdenFS will throw
errors at the user when such file are encountered, let's simply warn, and
continue.
Reviewed By: chadaustin
Differential Revision: D25430523
fbshipit-source-id: 4167b4cd81380226aead8e4f4850a7738087fd95
Summary:
Debug output belongs to stderr.
This makes it possible to turn on debug output without breaking programs
parsing stdout.
Reviewed By: singhsrb
Differential Revision: D25315954
fbshipit-source-id: c7813a824fbf6640cb5b80b5ed2d947e7059d53e
Summary:
From wez's description (slightly edited):
a symlink direction resolves outside the repo, and the new validation logic doesn't like that. the test is doing a redirect add bind then redirect add symlink. The second add is intended to implicitly delete the first one if the config is different, but it's not smart enough to realize that that is happening.
This adds an explicit `redirect del` before we do our second `redirect add`
Reviewed By: xavierd
Differential Revision: D25405011
fbshipit-source-id: 107fd272bbe830d8b23e437286ced00460902d91
Summary:
Stop implementing the legacy glob() Thrift function, since it's
deprecated, and just noise at this point.
Reviewed By: xavierd
Differential Revision: D25247641
fbshipit-source-id: a022fee169ad54c886d8f300b57bef233453fe8b
Summary:
We used to produce a confusing error message during glob evaluation
when . or .. was specified as a glob component. Instead, fail early,
with an error message that more directly explains the problem.
Reviewed By: genevievehelsel
Differential Revision: D24969096
fbshipit-source-id: fe70a8f4db1fdce8eec13890d20913b63a516518
Summary:
Be more specific about which PathComponent string failed to validate
in order to help diagnose downstream issues like glob syntax errors.
Reviewed By: genevievehelsel
Differential Revision: D24966004
fbshipit-source-id: cd3bc0aeaeb389caa13c86b91149d48c5afdb306
Summary:
I'm not entirely sure why these started failing, but enabling ui.allowmerge
made these run again.
Reviewed By: chadaustin
Differential Revision: D24697462
fbshipit-source-id: ec5ca987e7116edb12658eb7b4d03f1cf0f876d3
Summary:
Replace the old implementation of debugInodeStatus with the more
general traverseObservedInodes functionality, and add the ability to
customize its results with flags.
Reviewed By: xavierd
Differential Revision: D24300122
fbshipit-source-id: 0fbd3aa02575faa515fd7852441547d7de13426d
Summary:
We want to be able to fetch prefetch profiles on pull. That means we will need
to prefetch the contents of prefetch profiles for commits that we are not
currently on. Thus globFiles (the thrift endpoint used for prefetch profiles
fetching) needs to be able to take commit hashes to match and fetch against.
Why fetch prefetch profiles on pull? This would get the prefetch started earlier so
the files are hopefully fetched by the time the user needs them.
Reviewed By: chadaustin, genevievehelsel
Differential Revision: D23858659
fbshipit-source-id: 123e423d5117274b92405dbb5c2df690298a1c18
Summary:
On Windows, computing the sha1 of a materialized file requires opening up the
file in the working copy, as the file is cached there. Interestingly, this
potentially means that for computing the sha1 of a file, EdenFS may receive a
callback from ProjectedFS about that file or a parent directory. At this point,
EdenFS just refuses to serve this callback, as doing so may trigger an infinite
loop, or simply deadlocks. While this may sound weird, recursive callbacks are
not expected, as this signify that EdenFS view of the working copy doesn't
match what it actually is.
To close the loop, and from a code perspective, this means that computing the
sha1 of a file can fail and can throw an exception. Unfortunately, the code
didn't reflect this fact and exceptions were simply ignored, when that happens
during a checkout operation, this can leave the working copy in a weird state,
further agravating the mismatch between EdenFS view of the working copy, and
what it actually is.
Reviewed By: wez
Differential Revision: D24282048
fbshipit-source-id: 745af03189fe345150f0b1792ee1b37a1b8fb0d4
Summary:
While on Linux these can't fail (or, to be more precise: it doesnt' matter),
they can on Windows. One such exemple is when a user lock a file and triggers
an update that modifies this file. The invalidation will fail, and thus the
update should keep track of that file not being updated properly.
Previously, the invalidation would raise an exception, but that proved to be
the wrong approach as some state would need to be rolled back which the
exception didn't help in. For that, let's just return a Try and make sure that
we handle all the cases properly.
Reviewed By: chadaustin
Differential Revision: D24163672
fbshipit-source-id: ac881984138eefa65c053478a160e2a653fd3fdf
Summary:
I observed an integration test fail because Mercurial aborted with an
error about not handling EINTR, but it had no traceback. Always run
Mercurial with --traceback to try to debug these.
Reviewed By: xavierd
Differential Revision: D24204308
fbshipit-source-id: 44960bc645e5f97f61761e511b372328430fcda7
Summary:
Formats a subset of opted-in Python files in fbsource.
Black formatting was applied first, which is guaranteed
safe as the AST will not have changed during formatting.
Pyfmt was then run, which also includes import sorting.
The changes from isort were manually reviewed, and
some potentially dangerous changes were reverted,
and the directive was added to those
files. A final run of pyfmt shows no more changes to
be applied.
Reviewed By: zertosh
Differential Revision: D24101830
fbshipit-source-id: 0f2616873117a821dbc6cfb6d8e4f64f4420312b
Summary:
Now that `hg whereami` properly reads the SNAPSHOT file on Windows, the doctor
tests properly detect that Mercurial and EdenFS disagree about the current
commit, thus we can enable the remaining 2 tests.
Reviewed By: genevievehelsel
Differential Revision: D23819924
fbshipit-source-id: 21be19aff913e5e485d72e8cd730e6851ecaba2e
Summary:
on macOS we cannot safely use `fork`.
This commit replaces the use of `fork` in the startup logger subsystem.
This was a little tricky to untangle; originally (prior to any of
the `fork` removal efforts in this diff stack), the startup flow was
to spawn a set of processes via fork:
```
edenfs (setuid)
\-----edenfs (privhelper, as root)
\------edenfs (daemonized)
```
The forked children take advantage of being able to implicitly pass state to
the child processes from the parent. That data flow needs to become explicit
when removing the fork which makes some things a little awkward.
With fork removed:
* `edenfs` unconditionally spawns `edenfs_privhelper` while it has
root privs and before most of the process has been initialized.
* That same `edenfs` process will then spawn a child `edenfs`
process which starts from scratch, but that which needs to
run as the real server instance
* The original `edenfs` instance needs to linger for a while
to remain connected to the controlling tty to pass back the
startup state to the user, before terminating.
This commit deletes the check that `edenfs` is started originally
as root; previously the logic relied on the forked startup logger
continuing past the `daemonizeIfRequested` call and simply deferring
the check until after folly::init. With these changes we can't
easily perform such a check without adding some extra gymnastics
to pass the state around; the place where that is checked is in
the spawned child of the original edenfs, which is not a privileged
process and doesn't know the original euid. I don't believe this
to be a great loss as we tuck `edenfs` away under the libexec dir.
Reviewed By: chadaustin
Differential Revision: D23696569
fbshipit-source-id: 55b95daf022601a4699274d696af419f0a11f6f2
Summary: This diff fixes the eden redirection tests so it runs on Windows.
Reviewed By: xavierd
Differential Revision: D22958766
fbshipit-source-id: 45d26587831ed74d6bd7912b22c7c955b077f571
Summary:
tpx doesn't support heavyweight tags or rate limiting, and integration
tests regularly fail with timeouts on my devbig, so bump the process
start and process stop timeouts.
Reviewed By: genevievehelsel
Differential Revision: D23553924
fbshipit-source-id: fa9b8710395d61b087963d18718137e4525ae03d
Summary:
30 seconds is not enough time on heavily contented systems, including
CI. Bump the shutdown timeout to 120.
Also, correctly send SIGKILL to the daemon process when it's been
started with sudo.
Reviewed By: simpkins
Differential Revision: D22422784
fbshipit-source-id: dc7be0962705f1feb9643990309f570e352b68a0
Summary:
This test can't be non-flaky, because it relies on the kernel deciding
when to drop inodes from cache, and we've investigated it multiple
times. Given it tests a rarely-used function that would be better
expressed as a unit test in C++, just remove it for now.
Reviewed By: wez
Differential Revision: D23665455
fbshipit-source-id: 522e47113857eff399be4f2bb60e26e801d61e9a
