Summary:
fbcode_builder doesn't currently know how to build a Python extension module,
which means the Windows oss build fails on trying to load ntapi if we don't
pass the `--no-tests` flag to getdeps. Ideally we add extension module support
to the builder, but for now let's just disable the test case if the extension
module is unavailable.
Reviewed By: chadaustin
Differential Revision: D44804035
fbshipit-source-id: 10da54f6ea7d3699ac42421ee5cf57d883509bdd
Summary:
Eden's readdir and getfileAttribute endpoint returns an error when an entry in
a directory has a type that is not: regular file, directory, or symlink.
This causes issues for Buck2 because it propagates this error to the user. For
Buck2 a directory having a file that isn't a regular file, directory, or
symlink isn't an error case, it's just a file Buck2 wants to skip over. Buck2
would like to be able to differentiate real errors getting the filetype (like
say a network error) and having a weird file in some directory.
From chatting with Thomas, Buck2 is unlikely to ever care what type the file
is (if its not a file, dir or symlink). So it's sufficient just let buck2
know it's some "other" type of file. I think it makes sense to just add a non
source control type here. I also considered adding dtype as an attribute, but
I don't think we need it, but we could add that too.
In some cases it can be dangerous to add values to thrift enumeration
(SourceControlType enum we change below)
(reference post: https://fb.workplace.com/groups/thriftusers/permalink/785884732120941/).
But in our case, rust + Buck2 handles new enum types gracefully
(and with exactly the behavior we want):
https://our.intern.facebook.com/intern/diffusion/FBS/browse/master/fbcode/buck2/app/buck2_common/src/io/eden.rs?lines=157
so adding a value to the enum is safe (for buck2).
hack is our other client. they are going to handle it less gracefully:
https://www.internalfb.com/code/fbsource/[65673fd318750984372aeb5b44036a259a0d85d2]/fbcode/hphp/hack/src/facebook/hh_distc/package/package.rs?lines=441 but from what I can tell hack would also
error if they tried to list a directory with a socket in it with out this
change. Will confirm with them that this change is ok with them.
Reviewed By: chadaustin
Differential Revision: D44794698
fbshipit-source-id: 4e3ab7964fa2c0932b0363fb9ad62f24af74480c
Summary: Buck1 is EOL and these tests are thus no longer necessary.
Reviewed By: chadaustin
Differential Revision: D44717980
fbshipit-source-id: bfe0d9977243c35405e1b5cc988b687369488d0c
Summary:
On Windows, the `NtQueryDirectoryFile` low-level API exposes some behaviors—such as restarting partway through a directory enumeration, setting filename filters, and requesting single entries at a time—that are implemented by our ProjFS provider, but were not previously covered by our integration tests.
This change adds coverage by exposing `NtQueryDirectoryFile` via a Python extension module, then exercising it on an EdenFS mount in a new test.
Reviewed By: xavierd
Differential Revision: D44356956
fbshipit-source-id: 4114a0be95092b8276156ba7fd895f64d9e64c3a
Summary:
Now that EdenFS can fetch blob metadata from the server, let's make sure to
plug it to the `eden debug blobmeta` command.
Reviewed By: kmancini
Differential Revision: D44186943
fbshipit-source-id: a64f1384cf312e3c677505c330cfc82469fb83f3
Summary:
The external OSS build is broken because thrift has a really long path and
cargo/git on Windows do not lick such long paths.
I'm not fixing that. But the external OSS build is the only one we run in CI any
more. In the meantime the internal build has broken because no one has been
watching.
There were 4 different breakages.
Reviewed By: chadaustin
Differential Revision: D44189633
fbshipit-source-id: 2eedbc2b3bbf5d1def075d99f11f2273dbb1f4ab
Summary:
Fixing the asciitransform issue, unblocks a lot more tests.
This enables all the ones we know that were blocked by this.
Reviewed By: mshroyer
Differential Revision: D43966515
fbshipit-source-id: dfd988f81ec9b931f2d3bc6f34ad4ad82c2f2a61
Summary:
These tests seem fine? Maybe someone already fixed them?
But even with stress runs I can't get them to fail. I'll try on my intel too
to make sure it's not that.
Reviewed By: genevievehelsel
Differential Revision: D43986058
fbshipit-source-id: 7e3702a7efbd4637efbb235e626b421e9a3d28a1
Summary:
The takeover tests are failing in a couple ways.
First, there are failures:
multiprocessing seems to behave differently on mac than linux.
The process calls cause locking issues when "pickling". multiprocessing seems
kind un reliable, and we don't really need it in either of the used places.
Second. there are timeouts:
accessing an fd that was open before takeover seems to hang sometimes.
I can not manually repro on my M1, but don't have time to dig in right now, so
I will just leave a comment with some info on the issue and leave these disabled
for now.
Reviewed By: mshroyer
Differential Revision: D44000288
fbshipit-source-id: 76ef085967a495ffd3ab0a8aae337960368d75e0
Summary: In the next diff, we'll use this util function to acquire an exclusive handle to a file for a prjfs integration test.
Reviewed By: xavierd
Differential Revision: D44349708
fbshipit-source-id: ded7037266f539c681569f0c90f875717760fed5
Summary:
This one slipped through D44263797 and allows for integration tests to be run
with Buck2 on macOS.
Reviewed By: fanzeyi
Differential Revision: D44315942
fbshipit-source-id: d2de0773ba68f13fca9e8d5c067b82653646c757
Summary:
We've seen several cases where the sqlite database is corrupted causing EdenFS
to fail to start and requiring manual remediation. On Windows, we can always
reconstruct the sqlite database from scratch due to FSCK being able to build it
from scratch. Thus, we can simply delete the database on disk and continue
starting up.
Reviewed By: chadaustin
Differential Revision: D44155034
fbshipit-source-id: de05c814796ab8f76fd3cd9a3e98df438431c657
Summary:
It turned out `split_test` is failing for the same reason as other hg tests.
The assert raise hid the actual error so it wasn't very clear. So this diff adds some better reporting for that as well.
Reviewed By: kmancini
Differential Revision: D43962607
fbshipit-source-id: 137016260f286ee9577576d90e9e4372d4db960e
Summary: Most of these tests are already passing, except few of them are having issues with `_asciitransform`.
Reviewed By: mshroyer
Differential Revision: D43960522
fbshipit-source-id: b3bae8d3df1acc6f9b32057367309aff44c93de3
Summary: After the `._` fix, these tests are passing and we can enable them.
Reviewed By: chadaustin
Differential Revision: D43931347
fbshipit-source-id: c568eeaff5802901a20036c81faac58e44b74820
Summary: Some of these stats counters don't exist for NFS/PrjFS mounts. I filed tasks (T147665665, T147669123) to add these counters in the future, but until then we will disable some tests.
Reviewed By: kmancini
Differential Revision: D43963755
fbshipit-source-id: 4fd9ab9fa5bc123a6d24c669b655bfb18cf3a0a5
Summary:
This is another case where we had issues with the long default temp path on
macOS. Here, fake_edenfs was failing with SIGABRT on startup due to trying to
open a unix-domain socket with too long of a path.
D43925614 addressed this for tests based on EdenTestCase, but this doesn't help
with StartFakeEdenFSTestBase which only inherits from EdenTestCaseBase. So
this diff moves the environment variable override one level up in the type
hierarchy to cover more test cases.
Reviewed By: kmancini
Differential Revision: D43988120
fbshipit-source-id: 9eca299c42aea819d3e93ed5bb408b4a4783d8ee
Summary:
On macOS/NFS, for whatever reason, calling "unlink" on a directory will always
give a EPERM error. EdenFS is not involved in the error returned.
Reviewed By: fanzeyi
Differential Revision: D43962767
fbshipit-source-id: d0834b472be29657c1e37062a955337091a58be4
Summary: These are passing with no changes, likely fixed as a result of another diff.
Reviewed By: chadaustin
Differential Revision: D43952630
fbshipit-source-id: 9de8d375f21e33939339e72c7c08d3ba9abef070
Summary:
These were failing due to in_proc_mounts trying to open /proc/mounts. On macOS,
this file doesn't exist. We can however use the `mount` command instead to
achieve a similar goal.
Reviewed By: MichaelCuevas
Differential Revision: D43932260
fbshipit-source-id: dcdc707732196f2dcf4466acffba34deb3a86d25
Summary:
hg grep under the hood is `hg st | xargs grep`. The return code is the xargs
return code. xargs on mac and linux behave a little differently.
according the the linux man page:
```
EXIT STATUS
xargs exits with the following status:
0 if it succeeds
123 if any invocation of the command exited with status 1-125
```
according to the mac man page:
```
EXIT STATUS
The xargs utility exits with a value of 0 if no error occurs. If utility cannot be found, xargs
exits with a value of 127, otherwise if utility cannot be executed, xargs exits with a value of
126. If any other error occurs, xargs exits with a value of 1.
```
So a failure on linux in grep will result in a 123 code while on mac it will
result in a 1 exit code. I wish mac behaved like linux, but it doesn't. So lets
teach the tests to accept the xargs return code based on the platform.
Reviewed By: mshroyer
Differential Revision: D43924480
fbshipit-source-id: 0e6c2ec2c6a7553a1aed7c072f0464e5d1fa64c7
Summary:
This is theoretically fixable, but it seems like it would require a bunch of work and changes to test teardown.
For now, let's document why it doesn't work and provide a starting point if someone wants to fix it in the future.
Reviewed By: fanzeyi
Differential Revision: D43962277
fbshipit-source-id: b90f832a305ac7019f9031ddf3bee78a2e1e7817
Summary: This test passes on both architectures.
Reviewed By: genevievehelsel
Differential Revision: D43953611
fbshipit-source-id: 8c3477e6bc74a34cc437b17083967ba295ed4a6d
Summary:
On macOS, ._ (AppleDouble) files are sprinkled all over the place which
surprises engineers as they don't expect these files. We can easily detect when
such files are created and deny if EdenFS is configured to not allow them.
Reviewed By: chadaustin
Differential Revision: D43931001
fbshipit-source-id: db056624476dff15ce9b2f837d276244bc541667
Summary:
There are 2 reasons why this test is disabled:
1) `eden chown` crashes on macOS
2) Even if `eden chown` worked on macOS, it would require passwordless `sudo` which is not available on macOS Sandcastle hosts where we run our contbuild and diff time tests.
Since #2 is a big issue that we can't work around, let's disable the test.
Due to the test war room, I won't be fixing #1 in this diff. However, I created T147554174 to track fixing it.
Reviewed By: fanzeyi
Differential Revision: D43926151
fbshipit-source-id: f1572d2b876d45affdcdebf9b567e23c7730076a
Summary:
The basic integration tests were failing as they were trying to pass
--eden_logview to EdenFS, which didn't exist.
Reviewed By: fanzeyi
Differential Revision: D43929344
fbshipit-source-id: 0475a023adb8bc5ef2739764bed0cfe0dabea1ba
Summary:
On macOS, Python's `tempfile.mkdtemp` defaults to a long path under
`/private/var/folders`. `test_status_socket` creates a Unix-domain socket within
an Eden mount, but because of the long path this currently fails on macOS.
There are a couple different places where `mkdtemp` is invoked in our
integration tests, so here we override the default path on macOS to `/tmp`, by
way of the `TMPPATH` environment variable.
Reviewed By: genevievehelsel
Differential Revision: D43925614
fbshipit-source-id: fdd59668fcfe68ba57fabd0a18ec14bb1b3079d6
Summary: These tests are actually just testing code in `eden.integration.lib.linux` which solely used in these tests.
Reviewed By: xavierd
Differential Revision: D43924482
fbshipit-source-id: 167974cbd7c29396b00cc7394acc7e09891d5ce4
Summary: I missed this in the refactor of TreeOverlay -> SqliteCatalog
Reviewed By: fanzeyi
Differential Revision: D43854356
fbshipit-source-id: b192dde0d9c17f1bbeb19719f6fd5e7b25b1438b
Summary:
Currently, fsck will create an inode if a file is missing from disk, but present
in the source control tree. This matches the behavior of what PrjFS will
automatically do for files with in placeholder directories. But it doesn't match
for full directories.
Here I am teaching fsck to handle remove files in full and placeholder
directories differently. This change ensures that EdenFS's internal state always
matches that of disk after restart.
Notably this does not prevent files from reappearing after they are deleted.
If the parent directory is a placeholder and a file is deleted on disk while
eden is stopped, that file will come back after eden starts.
Ultimatly I would like all removals to behave the same: removed files stay
removed.
But preventing PrjFS from bringing back files in placeholder directories is
difficult. So we are starting with correcting eden's behavior for full
directories.
The issue is drawn out in this doc: https://docs.google.com/document/d/1kAlmIMAdqql6_zWQci2MVDH3vH3D7 (644ec66f6f)BeSuie-1s7RZmc/edit#heading=h.752ewec9aw22
along with the solutions considered.
This diff lines up with Solution 3. My plan is to pursue a full Solution 4 next to
fix the placeholder file reappearing thing.
Reviewed By: xavierd
Differential Revision: D43379227
fbshipit-source-id: 7da869236bdaf942c278803b4fa1565e8e4e98ee
Summary:
Currently, fsck will create an inode if a file is missing from disk, but present
in the source control tree. This matches the behavior of what PrjFS will
automatically do for files with in placeholder directories. But it doesn't match
for full directories.
This diff enumerates all the cases for removals while eden is running.
All the cases are listed out in this doc: https://docs.google.com/document/d/1kAlmIMAdqql6_zWQci2MVDH3vH3D7 (644ec66f6f)BeSuie-1s7RZmc/edit#heading=h.752ewec9aw22
There are 4 "failing" cases. There are really two types of failures. The first
type of failure is EdenFS being out of sync from the filesystem. The second type
of failure is both eden and the filesystem bringing back a deleted file.
Reviewed By: xavierd
Differential Revision: D43379228
fbshipit-source-id: a5a9d3e285d1014479fc492a817a0fac9e7fe351
Summary:
Currently, fsck will create an inode if a file is missing from disk, but present
in the source control tree. This matches the behavior of what PrjFS will
automatically do for files with in placeholder directories. But it doesn't match
for full directories.
To ensure the inode state matches the on disk state after fsck we need to detect
placeholder vs. full directories.
Fortunatly, the recall attribute seems to do just that.
This diff asserts the assumptions we are making about directorys. So that we
can safely use the recall attribute to distinguish placeholder and full
directories above.
In the above diffs, I will teach fsck to handle remove files in full add
placeholder directories the differently, so that the inide state matches disk.
Ultimatly I would like it all to behave the same: removed files stay removed.
But preventing PrjFS from bringing back files in placeholder directories is
difficult. So we are starting with correcting eden's behavior for full
directories. Currently, fsck will create an inode if a file is missing from disk, but present
in the source control tree. This matches the behavior of what PrjFS will
automatically do for files with in placeholder directories. But it doesn't match
for full directories.
Here I am teaching fsck to handle remove files in full and placeholder
directories differently. This change ensures that EdenFS's internal state always
matches that of disk after restart.
Notably this does not prevent files from reappearing after they are deleted.
If the parent directory is a placeholder and a file is deleted on disk while
eden is stopped, that file will come back after eden starts.
Ultimatly I would like all removals to behave the same: removed files stay
removed.
But preventing PrjFS from bringing back files in placeholder directories is
difficult. So we are starting with correcting eden's behavior for full
directories.
Reviewed By: xavierd
Differential Revision: D43379229
fbshipit-source-id: 0920834980f6ea6e76214555c196f355a847b8b1
Summary:
Over the past few weeks, we discovered that invalidating the working copy
without looking at the atime of files can lead to undesirable behavior due to
races between invalidation and placeholders being laid on disk (D42694759 (f308c20680)). We
also learned that in some cases, invalidation can take a really long time in
cases where very large parts of the repository is loaded, during which `eden
doctor` is stuck waiting for the invalidation to complete.
To solve the first one, we simply need to have `eden doctor` pass a non-zero
age, in this case since atime's granularity is 1h, we use 1h as the age. For
the second one, the Thrift handler is modified to allow backgrounding the
invalidation and `eden doctor` uses this new flag.
Reviewed By: kmancini
Differential Revision: D42785599
fbshipit-source-id: 9d86686f791e124b016685e3669004338ca33359
Summary:
Migrate off using legacy filelog repo and bundle2 protocols to modern
protocols. This fixes the test.
Reviewed By: muirdm
Differential Revision: D43517041
fbshipit-source-id: c997bd9863b849b4410a3fc163b67d6ec6b98dbd
Summary:
Migrate off using legacy filelog repo and bundle2 protocols to modern
protocols. This fixes the test.
Reviewed By: muirdm
Differential Revision: D43516085
fbshipit-source-id: f24ac78dbeb66192d4580fd6f74fb38a833d09d4
Summary:
See the previous diff for context. Basically, it's not a valid setup to be both
a shallow (remotefilelog) and an eager repo. Detect that and avoid it.
Reviewed By: muirdm
Differential Revision: D43516086
fbshipit-source-id: 13b6eb41097478f1e5fa16c9282709c17f5d4ade
Summary:
`hg init` by default creates the "filelog" repo, which is a tech-debt
that we never use in production. Add a way to pass configs to `hg init` to
create different kinds of repos.
Reviewed By: xavierd
Differential Revision: D43516087
fbshipit-source-id: 9be96317883a61be8f958d1eb88f6803c81e752a
Summary:
Our fsck does not properly handle renamed placeholders: renamed files
do not need to be present in scm and they may not be full or hydrated.
For example: clone a new repo, move a file. That file will just be a
placeholder. But eden materializes all renamed files. So the inode will
be materialized, but the contents will not be present on disk.
This works out ok, because when ProjFS reads the file, it will send a
"read" request for the old path. And EdenFS serves read requests
directly out of the source control state.
Our Fsck assumes that all files must have their contents present on
disk or exist in source control. Which generally make sense since
file contents should be read from disk or source control. But renamed
files are an exception.
Generally, this materializing renamed files without them being at least
hydrated when we gernally have a rule in windows eden that this should
not be the case, does feel kinda hacky. But we need to materialize them
for status and diff I think. So the option is to force hydrate things as they
are renamed. And this would mean unnecessarily fetching contents.
So I am not sure that its worth it.
We can at least fix this fsck issues for now, by fsck-ing renamed files
like populated & materialized ones. The inode should be materialized,
and we need to ensure the inode is the correct type. This diff teaches
fsck to detect renamed files and fix them up like materialized ones.
I discovered another fsck bug while fixing this. When files are removed
while eden is stoped, Eden will keep them in the overlay, but does not
add them back to disk. So the file will be removed from disk, but present
in the overlay. hg status does not match disk in this case ...To fix in a later
diff.
Reviewed By: xavierd
Differential Revision: D42268664
fbshipit-source-id: cb36a6513bb30a1c1951b5c8cc0ca66f78edbb67
Summary:
Andrew Krieger reported that eden doesn't seem to notice when files are
truncated on windows. And it repros pretty easily:
```
fbclone fbsource tmpfbsource
cd tmpfbsource/arvr/libraries/avatar/Libraries/asset/items/gltf
Clear-Content src/GltfCompactSkinningData.cpp
hg st
# empty
```
Eden is getting a file closed and NOT modified notification. Microsoft has
confirmed that this is unexpected and would fix it.
But in the meantime there is something we can do to work around it. We do get
the notification "pre convert to full". Full files should be materialized in
EdenFS, so we can use this notification to mark the inode as materialized early.
Handing notifications is async, so this should not effect performance.
Benchmarking creating a lot of files, shows that this doesn't seem to make much
of a difference.
We will roll this out with a config, and then roll it back once microsoft's fix
rolls out.
Reviewed By: xavierd
Differential Revision: D42465547
fbshipit-source-id: cc41fa8c7ad7fbf96ae4a9af5b3a7e6f477fc449
Summary: Now that we are using disk image redirections on macOS again for integration tests, we should be able to freely enable integration tests on Sandcastle and avoid the APFS subvolume limit issues.
Reviewed By: chadaustin
Differential Revision: D42468402
fbshipit-source-id: bca888f805bc01ddf13300b5b96dc10fe9e5c924
Summary: Now that disk image redirections are available to use again, we should move integration tests back to them. This will allow us to bypass APFS subvolume limitations on macOS Sandcastle hosts and start running integration tests on Sandcastle.
Reviewed By: kmancini
Differential Revision: D42468376
fbshipit-source-id: e0253ef003a191bdf4595772461a3870db613422
