Summary:
The restart tests have started failing recently due to various timeouts. The
root cause is well understood to be caused by an increase in the dependency
graph of EdenFS. Let's increase the timeout while we figure out a way to
decrease the dependencies.
Reviewed By: genevievehelsel
Differential Revision: D35829901
fbshipit-source-id: d7e03ea68308b9eb36e3584d3c073b41378a78de
Summary:
In Mercurial terminology, a `hg reset` operation simply updates the working
copy parent but doesn't affect the state of the repository. A diff/checkout
operation however would compare the on-disk state of the repository with the
working copy parent. Thus EdenFS must respect this behavior.
On Linux and macOS, a reset operation merely updates some state in the
EdenMount, but since these platforms reads data from the inode hierarchy, the
reset operation doesn't affect future reads to the repository. These would
still read the data from the previously checked out revision.
On Windows however, ProjectedFS requires us to read data from the currently
checked out commit, and thus we bypass the inode hierarchy. The downside is
that a reset operation changes the RootId in the EdenMount which is used to
walk Mercurial's tree hierarchy. What this means is that a read operation
performed after a reset would read the content of the file as it is in the
reset revision. This is not the behavior that Mercurial expects.
To solve this, we need to keep track of both the last checked out revision to
be used in response to read callbacks and the last reset revision, to be used
in diff operations. To achieve this, we can use the newly introduced SNAPSHOT
file format.
Reviewed By: fanzeyi
Differential Revision: D35293079
fbshipit-source-id: 5cfae510db8bc759d51447e6c24f021c104b7353
Summary: Do the same trick we for some other sparse commands to allow usage in eden repos.
Reviewed By: StanislavGlebik
Differential Revision: D35506423
fbshipit-source-id: b02f8f76f18acbbd24431da0d41b8322b9556658
Summary:
In order to properly support reset on Windows, EdenFS needs to keep track of
both the checked out revision, and the reset one. Since this state needs to
persist across EdenFS restarts, we'll need to store it in the SNAPSHOT file.
This diff merely adds support for this new SNAPSHOT format, but doesn't write
it.
Reviewed By: chadaustin
Differential Revision: D35448641
fbshipit-source-id: 59fbafdadc885dfec8efd119c4fe1ff582987af0
Summary:
We have a few error reports of
```
Reviewed By: xavierd
Differential Revision: D35264311
fbshipit-source-id: 4a601a8b8da7aa89eede94060105ebffaf777e83
Summary:
We have a few error reports of
```
Reviewed By: xavierd
Differential Revision: D35262460
fbshipit-source-id: 4b9ceea8e275c97aeca25a1dad2f286f0311e547
Summary:
During a checkout operation, the server state executor is overwhelmed with
requests, with some of them on Windows waiting to take some overlay locks.
Unfortunately, reading the root tree involves several round trips to the same
executor which get queued after the checkout requests. In practice, what this
means is that a request to read the root tree while a checkout operation is
ongoing can be waiting for a very long time and since every ProjectedFS
operation starts by obtaining the root tree, these can block user commands for
a extended amount of time, giving the feeling that EdenFS is hung.
To solve this, we can simply hold a reference to the root tree in the
EdenMount, this root tree is only going being updated during a checkout or a
reset operation, and we can thus easily obtain the root tree then instead of
for every ProjectedFS operations.
Reviewed By: chadaustin
Differential Revision: D34831590
fbshipit-source-id: 7083b4bdfc72c23d8fe400411e31ad66d2d72624
Summary:
When a directory is renamed, EdenFS receives a single notification on that
directory. Unfortunately, due to the implementation of the fileNotification
function, this means the old directory would get removed, and the destination
directory would just get created. However, none of the files or other
directories would get added to the inode hierarchy.
To fix this, we need to make sure to recursively add the content of a directory
when it gets created as some files may be present in it.
Reviewed By: fanzeyi
Differential Revision: D34731270
fbshipit-source-id: 39b25441408c1e44ca7e62522699b053dd3e6c5d
Summary: The invocation of `eden doctor` within rage is causing `eden rage` to hang on Windows when an existing eden mount is hanging. See T113845692 for more info.
Reviewed By: xavierd
Differential Revision: D34741692
fbshipit-source-id: 90d3ed17cec90c8d7b00e9e8315d6d99d21ff83d
Summary:
Several tools rely on the .eden directory to be present to detect EdenFS.
Unfortunately, removing it by hand would make it disappear breaking these
tools. Thankfully ProjectedFS is providing ways to deny some rename/deletion
from happening.
Reviewed By: chadaustin
Differential Revision: D34433910
fbshipit-source-id: 4d37a6d9eeea3f16f4e1401beac20ef6d6dce71d
Summary:
Some of EdenFS's integration test functionality can build on macOS, so
implement the required tweaks.
Reviewed By: kmancini
Differential Revision: D34291419
fbshipit-source-id: 296f077f4a3311d7fc8f04ed705fd06019e34212
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/106
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/107
Summary
* Add EdenFS builds on external CI now EdenSCM is good
* Mac builds on github actions by using brew for system dependencies
To make this work had to fix some path ordering issues with install directories for Linux and Mac, and generalise the homebrew path fixups we were doing for bison to all the used homebrew packages.
Previously Installed packages were being added after system paths, so our own installed thing might be ignored. On github these meant system python 3.9 was being used for hg tests rather than our specified 3.8 (this showed we have some test fails on python 3.9 with "SystemError: deallocated bytearray object has exported buffers", that are beyond the scope of this diff to fix)
Also needed to include the getdeps generated python into the generated edenscmdeps3.zip archive setup.py produces otherwise EdenFS tests failed to import thrift.Thrift
Eden tests are hanging when run externally about half way through, so disable them on github actions for now as this PR is already fairly large. They work when run locally on an internal devserver, so probably some bit of environment necessary is not defined in the test runner
Reviewed By: chadaustin
Differential Revision: D34116505
fbshipit-source-id: d0d628db5daabc28d0bd8997cd5c1bc885ed1e73
Summary:
Convert type comments to type annotations.
Produced by running:
```
python -m libcst.tool codemod convert_type_comments.ConvertTypeComment eden
```
from a parent directory
Reviewed By: shannonzhu
Differential Revision: D34110680
fbshipit-source-id: 23081a8867b93cd349271075428176af514efa2b
Summary: Don't set the "fb" cargo feature via cmake unless we are in fb build.
Reviewed By: quark-zju
Differential Revision: D33781554
fbshipit-source-id: 39c61dc1631ebfd8a2c858b5e214c6a09ce0b005
Summary:
The current serialization type we use for the graceful restart protocol only
allows us to send mount specific information in the non error case.
If we want to send general takeover information, like the order of the file
descriptors we send along with the takeover message, we are out of luck.
I am changing the final type we send during takeover to be a struct so that
we can add non-mount specific information. This allows us to create an explicit
order of our file descriptors and will let us send a dynamic number of general
file descriptors.
Currently, we have to send all the expected general file descriptors
(thift socket, lock file, mountd socket) in a fixed order.
And those file descriptors have to be valid because we cannot send an invalid
file descriptor with sendmsg.
When NFS is not enabled we do not initialize the mountd socket so we cannot
send a valid filedescriptor for the mountd socket (unless we sent some dummy
file descriptor which feels really hacky and not a good permanent solution).
To be able to fix the bug we have in graceful restart due to not having
a mountd socket we need to be able to send only a portion of the general
file descriptors.
When NFS is not enabled, we do not initialize the NFS Server. This causes
a seg fault during takeover because we try to shutdown the nfs server.
We should only shutdown the NFS Server when we initialized it. This
means we only pass the mountd server socket during takeover when NFS was enabled
when we started the server.
To allow use to enable the NFS Server across graceful restart we only
perform a takeover start when the mountd server socket was received during
takeover.
Reviewed By: xavierd
Differential Revision: D33755447
fbshipit-source-id: cc471c3051de8e41c21a4d1d8e42d93d5295e90e
Summary:
In one of my next changes I am going to be introducing a integration test where
NFS is not enabled.
Previously, we did not have any tests that ran with NFS disabled and so a bug
slipped through the cracks in graceful restart when NFS is disabled.
If I am going to disable NFS then these tests should not pretend to run on NFS.
This introduces a parameter to the eden test decorator that allows skipping
the NFS tests.
Reviewed By: genevievehelsel
Differential Revision: D33247961
fbshipit-source-id: ab37bd1aa64b9b09a69fe5e90e8068d4f444cd29
Summary:
When an update removes a directory but a materialized ignored file/directory
was present, EdenFS will rightfully not remove that ignored file/directory.
However, the parent directory would have been a placeholder on disk. And since
placeholder directories can never be made full[0], the directory keep its
placeholder. Listing this directory would thus issue an opendir callback, but
EdenFS would fail due to that directory not being present in the manifest.
Since this case can only happen in this case, let's simply return an empty
directory and let ProjectedFS merge it with the on-disk directory content.
[0]: https://docs.microsoft.com/en-us/windows/win32/projfs/cache-state
Reviewed By: genevievehelsel
Differential Revision: D33676621
fbshipit-source-id: bdbf769175a6471a1a709bdb0b03a45bd60ef366
Summary:
Now that notifications are running in a serial executor, and are also issuing
disk IO, notifications are significantly slower than they used to be. While
writing to the working copy is an overall anti-pattern, some workflows (like
Unity) do and it's thus critical that their performance isn't affected
negatively.
In order to solve this, we can simply move the handling of notifications to the
background and answer the notification immediately, since notifications can no
longer fail, we shouldn't need to send an error back to ProjectedFS which would
anyway be ignored.
The drawback is of course that applications are no longer blocked while the
notification is being processed in EdenFS, and thus any operation that needs to
get a synced up inode hierarchy will need to wait on all the pending
notifications.
Reviewed By: chadaustin, genevievehelsel
Differential Revision: D32480993
fbshipit-source-id: 7ad6b07f540f7d9a52a35a0ff3b94911ef5267af
Summary:
With Facebook having been renamed Meta Platforms, we need to change the license
headers.
Reviewed By: fanzeyi
Differential Revision: D33407812
fbshipit-source-id: b11bfbbf13a48873f0cea75f212cc7b07a68fb2e
Summary:
It feels a bit weird for getDaemonInfo().status's meaning to diverge
from fb303's getStatus(), but fb303 is being restructured so remove
the custom getStatus implementation.
Reviewed By: praihan
Differential Revision: D33176576
fbshipit-source-id: 5a7b7f3394b123b1073191508c2c0c1ebcda5bb5
Summary:
The EdenApi methods requiring `reponame`. It leaks the HTTP URL detail to many
more layers.
This diff makes the `Client` use the repo name provided by the config, or
explicitly via the `repo_name` API (as in tests). The HTTP client uses the repo
name in its config so callsites don't need to provide the name again.
Previously, if reponame is not provided, the HTTP client builder will use
"unknown" as fallback. That behavior is changed to an error instead.
The above change breaks some tests using production configs on scmstore
construction. scmstore is updated to properly avoid constructing EdenApi
if repo name or URL is not provided, instead of constructing an EdenApi
HTTP client using the wrong "unknown" URL silently.
This diff does not change the `EdenApi` trait yet, since that is a large change.
Reviewed By: yancouto
Differential Revision: D32800595
fbshipit-source-id: a156e18b2f6fc41804cc40e3c926086be63fd9b9
Summary:
This option is only ever used for tests, let's remove it and simplify the code
a bit.
Reviewed By: singhsrb
Differential Revision: D33103247
fbshipit-source-id: e822de6dba541152452984f6aad5258a5585db02
Summary:
From the Thrift documentation:
* Ensure that all inflight working copy modification have completed.
*
* On some platforms, EdenFS is processing working copy modifications
* callbacks from the platform in an asynchronous manner, which means that by
* the time a write/creat/mkdir/unlink/etc syscall returns from the kernel,
* EdenFS may not have updated its internal state.
*
* Thus, an application making changes to the working copy and quickly
* requesting EdenFS to perform an operation on it will race with EdenFS
* updating its internal state and may thus get stale data.
*
* To avoid this, EdenFS queries need to internally synchronize the working
* copy before performing the query itself. This structure defines how EdenFS
* will do this.
*
* Applications that care about synchronizing EdenFS up to a certain point in
* time are expected to set a non-zero syncTimeout once to synchronize EdenFS
* and then issue all their thrift requests with a syncTimeout of 0.
Reviewed By: chadaustin
Differential Revision: D32900351
fbshipit-source-id: c242d881b2f2db6cbca544288a0eac5959469fc5
Summary:
The test started failing on CI due to the file not being read when being
locked. ProjectedFS behavior when the file is a placeholder vs a full file
differs when that file is locked and invalidated. In the case of a placeholder,
the file will be removed, while the full file won't. Let's force a full file by
simply reading it before locking it.
Reviewed By: kmancini
Differential Revision: D32776736
fbshipit-source-id: faf0bca1becc4bfed8cd6bc804e33a0540f3b8cb
Summary:
There was a subtle bug in D31782916 (87ef50135e) where a wildecarded enumeration may send
the same directory entry multiple times to ProjectedFS. This can happen if the
first directory entry returned to ProjectedFS is not where iter_ initially
points to, which is almost always the case when a wildecard is being used. Thus
the first call to advanceEnumeration will simply find the first element sent to
ProjectedFS and send it again on the next readdir call.
Reviewed By: fanzeyi
Differential Revision: D32742433
fbshipit-source-id: 12904a5d2df4407072b8d775950f92ea93344a73
Summary:
Now that takeover is implemented on nfs, we can enable the integration tests
for graceful restart on NFS.
This last test relies on file invalidation, so we implement this as fully as we
can on an nfs mount in order to enable this test.
Reviewed By: xavierd
Differential Revision: D32225243
fbshipit-source-id: c6ddbba54eb71edf6bfe70285a775d438a9d0acd
Summary:
Hardlinks aren't supported, but we don't have tests verifying this. While we
may support them in the future, let's make sure we don't break them until we
do.
Reviewed By: chadaustin
Differential Revision: D32480994
fbshipit-source-id: ae00f605a3cea8ebd4fbcba06098d109c0efa8e1
Summary:
Now that graceful restart is supported, we can enable the integration tests!
These are all passing. The IO test is a little flakey, but the fuse version is
as well. So I haven't invested a lot of time in it.
One test is only compatible with version 1 and 2 of the takeover protocol
which is not compatible with takeover for nfs mounts. So that test will remain
disabled.
Reviewed By: xavierd
Differential Revision: D31738202
fbshipit-source-id: 0160310355138db59a8e671d3888b948e896eefa
Summary:
Now that we have NFS mounts we need to support takeover for those mounts.
This commit teaches the takeover server and client to serialize and deserialize
NFS information.
NFS information includes:
- the mountd socket. Mountd is used to register mount points, and should keep
the same socket across graceful restarts.
- a connected socket for each of the NFS mount points.
- All the rest of the mount point information that we send for fuse mounts (with
the exception of the fuse negotiated parameters.
Future commits will teach the NFS components to graceful stop and yield this info
as well as restart with this info.
Reviewed By: xavierd
Differential Revision: D31802787
fbshipit-source-id: 667625589b77eed79f7e17390013a818d4be7068
Summary: Instead of fetching the file size and sha1 of a file separately, we use the new FileInode::getBlobMetadata() method to get them at the same time.
Reviewed By: xavierd
Differential Revision: D32401043
fbshipit-source-id: 7fcedeaf9070931e55d3fd8b23544b8b56118d1a
Summary:
Adds logic for new thrift endpoint that allows users to request attributes for a given list of files.
Requested attributes are obtained for each file and stored in a FileAttributeData struct. Each file is then mapped to a FileAttributeData struct (or an EdenError). If a requested attribute cannot be found for a given file, an EdenError is returned instead of a FileAttributeData.
Integration tests cover too many cases, so I've commented out the excess cases and will turn them into unit tests in a later diff. For now, they are available for use, but not included by default.
Integration tests cover the following cases:
- Requesting all attributes for 2 files
- Requesting attributes for a non-existent file
- requesting just file_size for a file
Commented out Integration tests include:
- Requesting attributes for empty string path
- Requesting attributes for a file that contains dot components
- Requesting attributes for a directory
- Requesting sha1 for a symlink (should file_size for a symlink also return an EdenError?)
- requesting all attributes for no files
- requesting no attributes for a file
- requesting just sha1 for a file
Reviewed By: kmancini
Differential Revision: D32179305
fbshipit-source-id: 6c23bb7780154159fdd9626d332c641311afbe57
Summary:
In every test setup, writing a file to the repository will always call `hg add`
in order to this newly added file to be added in subsequent `hg commit`. Since
`hg add` can be very expensive (in particular on Windows), some tests are
spending most of their time calling `hg add` instead of running the body of the
test. As an easy way to avoid this, let's stage the added files in memory
first, and then call `hg add` once just before `hg commit`.
Reviewed By: genevievehelsel
Differential Revision: D32333775
fbshipit-source-id: 8cc6cfad7ec73f6f0c50d7d1e6affa6cdd828e2f
Summary:
The new sparse profile config logic was triggering during an Eden code
path, where sparse doesn't exist. This caused a crash. Let's check for sparse
before trigging the path.
This only occured if there was another exception during checkout.
Reviewed By: quark-zju
Differential Revision: D32436881
fbshipit-source-id: 6d5da888e4f8a62a05a390c7e87bbc3061356fc2
Summary:
In order to compute the sha1 of materialized files in the working copy, EdenFS
on Windows will open and read the file out of the working copy directly. In the
case where EdenFS is wrong about the state of the file, this can lead to a
ProjectedFS callback to be triggered which may need to take the same lock that
the sha1 computation holds. This obviously results in a deadlock and to protect
against this, EdenFS would bail early when detecting recursive callbacks.
With read-only callbacks having been switched to only take into account the
underlying Mercurial data, and not the inode state, these callbacks will never
attempt to take any inode locks. This also means that the deadlock described
above can no longer happen: the ObjectStore layer will never attempt to read
anything from the working copy, thus recursive callbacks are impossible. These
checks are thus unecessary.
In theory, these checks could stay in place as a safety measure, but EdenFS
will soon need to list directory entry and call PrjGetOnDiskFileState in
response to file/directory creation/removal to handle the case where these can
happen out of order.
Reviewed By: genevievehelsel
Differential Revision: D32149411
fbshipit-source-id: 73b228bb8c73416b38f7f2042c25964e1aef3c4e
Summary:
In ProjectedFS terminology, the "backing store" refers to the state of files
and directories in the source control backend. For the longest time, EdenFS
assumed that the Inodes hierarchy was the "backing store", but this lead to
subtle bugs. For instance, renaming a non-hydrated placeholder would render the
file unreadable due to the read callback from ProjecedFS callback using the old
file name, not the new one
(https://github.com/microsoft/ProjFS-Managed-API/issues/68). It also meant that
files just created would always get a tombstone written on disk when being
removed as ProjectedFS queries EdenFS when removing a file via the access
callback. Returning true from this callback means that this file is present in
the backing store and thus ProjectedFS needs to create a tombstone on disk.
In order to fix these issues, EdenFS needs not to query the Inodes hierarchy,
but the Tree hierarchy. The latter matches with the expected behavior from
ProjectedFS.
For modification callbacks, these are by design acting on the Inode hierarchy
and thus are unchanged.
Reviewed By: chadaustin
Differential Revision: D32022639
fbshipit-source-id: bef18a0709a7eff9ef0a48a90126d1339be0023e
Summary: This should give us more diff time signal.
Reviewed By: chadaustin
Differential Revision: D32207108
fbshipit-source-id: c9f6bc947a2622021194f8fa3d673cd61ae9bb2a
Summary: sys.platform is "win32", not the path to python.
Reviewed By: fanzeyi
Differential Revision: D31554106
fbshipit-source-id: 64b388d2fb8a493f811a0cf22fe2471a25bfbf7e
Summary: This appears to have broken all the tests on Linux.
Reviewed By: zhengchaol
Differential Revision: D31505082
fbshipit-source-id: 610eb941d0f0eb536a4688ac2637a8466be0b05c
Summary:
On Windows, edenfsctl.real is a par file that Windows can't execute directly.
Thus let's have Python run it.
Reviewed By: fanzeyi
Differential Revision: D31477721
fbshipit-source-id: d5a699ceb3d3b1b3d5778ef5720bca7c299bed80
Summary:
These tests have been failing for a long time due to the expected output on
Windows containing \ but the test using the / separator.
Let's simply use os.path.join to build a path with the right separator in these
tests.
Reviewed By: fanzeyi
Differential Revision: D31477722
fbshipit-source-id: a4ac25750647229974e23e305508e83917011fef
Summary:
The goal of this stack is to remove Proxy Hash type, but to achieve that we need first to address some tech debt in Eden codebase.
For the long time EdenFs had single Hash type that was used for many different use cases.
One of major uses for Hash type is identifies internal EdenFs objects such as blobs, trees, and others.
We seem to reach agreement that we need a different type for those identifiers, so we introduce separate ObjectId type in this diff to denote new identifier type and replace _some_ usage of Hash with ObjectId.
We still retain original Hash type for other use cases.
Roughly speaking, this is how this diff separates between Hash and ObjectId:
**ObjectId**:
* Everything that is stored in local store(blobs, trees, commits)
**Hash20**:
* Explicit hashes(Sha1 of the blob)
* Hg identifiers: manifest id and blob hg ig
For now, in this diff ObjectId has exactly same content as Hash, but this will change in the future diffs. Doing this way allows to keep diff size manageable, while migrating to new ObjectId right away would produce insanely large diff that would be both hard to make and review.
There are few more things that needs to be done before we can get to the meat of removing proxy hashes:
1) Replace include Hash.h with ObjectId.h where needed
2) Remove Hash type, explicitly rename rest of Hash usages to Hash20
3) Modify content of ObjectId to support new use cases
4) Modify serialized metadata and possibly other places that assume ObjectId size is fixed and equal to Hash20 size
Reviewed By: chadaustin
Differential Revision: D31316477
fbshipit-source-id: 0d5e4460a461bcaac6b9fd884517e129aeaf4baf
Summary: This diff fixes some integration test errors after enabling the new edenfsctl.
Reviewed By: xavierd
Differential Revision: D30789741
fbshipit-source-id: 02d74defc41def4fb6ea0cc4694f944b4c0044e2
Summary: This diff adds the Rust edenfsctl to our open source build.
Reviewed By: xavierd
Differential Revision: D30788685
fbshipit-source-id: 603caa933ecfc5af0ede7e22f6c7911462da3a65
Summary:
We run periodic inode unloading for unlinked inodes on NFS because we get no
information from the client on when inodes are no longer needed, and we have to
clean them up at some point for memory and disk reasons. See previous commit
summaries for more details on this (D30144901 (ffa558bf84)).
Let's add some counters on this so we have a bit more visibility into the
process. This counter is meant to mimic the PeriodicUnloadCounter counter.
Reviewed By: chadaustin
Differential Revision: D30966688
fbshipit-source-id: cfc8d769b53073d9f4c0c27b6bee20e222c6c8d2
Summary:
It can be surprising when a job suddenly is no longer able to run sudo, or no
longer run as root that all the tests are marked as being successful, despite
the fact that they no longer run. Let's recognize when we run on EdenFS to
allow tests to fail if they can no longer run EdenFS.
Reviewed By: zhengchaol
Differential Revision: D30357402
fbshipit-source-id: c3758d7a5a3c575dd68bd97062ae24abe4124874
Summary:
Now that we might have multiple kernel protocols per mount (i.e. both fuse and
nfs on macOS) let's include them in eden rage.
Reviewed By: xavierd
Differential Revision: D31154042
fbshipit-source-id: 38e7630829d70fe9dd6dbeabacc3b538ee798e0d
Summary:
We periodically need to dereference inodes on NFS because we get no other info
from the kernel on when should dereference them.
This means the NFS kernel might have references to inodes after we delete them.
An unknown inode number is not a bug on NFS. It's just stale, so the error should
reflect that.
Reviewed By: xavierd
Differential Revision: D30144898
fbshipit-source-id: 3d448e94aea5acb02908ea443bcf3adae80eb975
Summary:
`eden prefetch` and `eden glob` return lists that despite being called
"maching files" actually contains both files and directories.
In some cases, we only want the list of files and it introduces unnessecary
overhead on our clients for them to have to stat all the files in the list to
filter out the dirs. Let's add an option to just list files.
Reviewed By: chadaustin
Differential Revision: D30816193
fbshipit-source-id: 6e264142162ce03e560c969a0c0dbbc2f418d7b9
Summary: the rage summary is getting hard to quickly parse, so this underlines each section header, as well as unifies underline looks (with `eden stats`). This adopts the underline code from `eden du` and makes it a util function for shareability.
Differential Revision: D30857773
fbshipit-source-id: 66b5b06f5b0125304d45d3465a8bc2248693b791
Summary: This test can fail for not having `USERNAME` environment variable when the test is run as root. Let's just skip the test when this happens because it doesn't make sense to drop priv as root.
Reviewed By: xavierd
Differential Revision: D30868518
fbshipit-source-id: 14ff6db218b1477f5905f2df3ad075a5ca186117
Summary:
This change has the unintended effect of causing any Thrift calls to
potentially issue a recursive EdenFS call due to symlink resolution requiring
running `readlink` on the root of the repo itself.
Fixing this isn't really possible, thus let's revert the change altogether, we
can force clients to issue a realpath before issuing EdenFS Thrift calls.
Reviewed By: kmancini
Differential Revision: D30550796
fbshipit-source-id: 9494c8e08c8af2392eeb344879f156cb56f93ea6
Summary:
The rate metric can be unreliable, and in some environment is never updated by
the time it is read by the test. Since this test cares about the number of
events, using count is a better metric, and more reliable.
Differential Revision: D30377128
fbshipit-source-id: f10c656567e3a3c07b66ecc6fc563a53199e3088
Summary:
In the ASIC test environment, we've see cases where this isn't set, let's
simply skip the test in this case instead of failing.
Reviewed By: chadaustin
Differential Revision: D30353929
fbshipit-source-id: 956da6f8f12de025b8ca72e40097f1f9d50e6bf7
Summary:
In the case where the path to the mount has symlinks, EdenFS would only accept
the path to it that was specified at mount time, even though another path may
refer to the same directory.
To solve this, we can simply normalize paths in all the Thrift endpoint to make
sure that EdenFS always refers to a mount point under its non-symlinked path.
Reviewed By: chadaustin
Differential Revision: D30320515
fbshipit-source-id: e578d059a3b1307d6b24c4b9bdb1ceb3b534c460
Summary:
The changes in this diff makes the code to "run" and skip correctly according to our rules.
All integration tests are now runnable on Windows with mode/win
Differential Revision: D30143255
fbshipit-source-id: b2ddbff7268f182274b3755f4b28df6ac6cdeef4
Summary:
This newly added config controls the default behavior of `eden clone` and
whether a new clone will use NFS or FUSE. This is intended to facilitate the
transition to NFS from FUSE on macOS.
Differential Revision: D30110056
fbshipit-source-id: ea6b493aa803297952b46434f6d11d8edf58e40b
Summary:
This adds the options to `eden stats` for collecting only fast stats and printing in JSON.
`eden stats` can be slow especially due to collecting fb303 counters and private bytes. An example use case of this new lightweight endpoint is that Buck can poll it to display Eden related info in its cli (see [post](https://fb.workplace.com/groups/132499338763090/permalink/210396380973385/) for context).
Reviewed By: xavierd
Differential Revision: D29687041
fbshipit-source-id: a663e71231527c5dfb822acbf238af0ac6ce4a00
Summary:
Per comment in D30017261 (e9c039ab4a)
> [...] we can get rid of this ESTALE check. That dates back to when edenfs had a bug that returned ESTALE when reading the .eden directory sometimes.
Reviewed By: xavierd
Differential Revision: D30024979
fbshipit-source-id: 645097c8f689c916245845561fd3d824ff7df8b4
Summary: 2 of the disabled are passing, and the last one just needed a simple tweak.
Reviewed By: fanzeyi
Differential Revision: D30029260
fbshipit-source-id: ad386632e8b6fbf9c76530c8ce2df7d38990bbbd
Summary:
The mode bits don't make a lot of sense on Windows, so let's use the plain
0o777 as that's what they will look like on Windows.
Reviewed By: chadaustin
Differential Revision: D30028892
fbshipit-source-id: ced737f1290438708536b732095cee18e1b1c073
Summary: The test is passing, enable it.
Reviewed By: chadaustin
Differential Revision: D30028894
fbshipit-source-id: b268fc63220704466c72c688a1f0f04a47573641
Summary:
The create_editor_that_writes_commit_messages function uses bash which is not
available on Windows. Instead, we can simply pass --message to Mercurial for
the same effect.
Reviewed By: chadaustin
Differential Revision: D30028893
fbshipit-source-id: 0a588c37bd4da85176546568126893192b29a574
Summary:
If Mercurial asks EdenFS to update to a commit that it has just created, this
can cause a long delay while EdenFS tries to import the commit.
EdenFS needs to resolve the commit to a root manifest. It does this via the
import helper, but the import helper won't know about the commit until it is
restarted, which takes a long time.
To fix this, we add an optional "root manifest" parameter to the checkout or
reset parents thrift calls. This allows the Mercurial client to inform EdenFS
of the root manifest that it already knows about, allowing EdenFS to skip this
step.
Reviewed By: chadaustin
Differential Revision: D29845604
fbshipit-source-id: 61736d84971cd2dd9a8fdaa29a1578386246e4bf
Summary:
We need to set the infinitepush path too so that the Mercurial autopull code
won't try to pull from Mononoke
Reviewed By: genevievehelsel
Differential Revision: D29943227
fbshipit-source-id: a67dbfe97e7ab46dee885d9ec91a4d194dc2bd37
Summary:
If for some reason EdenFS cannot be started, we shouldn't attempt to run the
fsck tests as these would always fail.
Reviewed By: genevievehelsel
Differential Revision: D29918436
fbshipit-source-id: 6e4a01a747157427e5c1028084e32cef8066c96a
Summary: These are needed for mercurial in the test cases, we set this in the testharness C++ repo as well
Differential Revision: D29868460
fbshipit-source-id: e11cf41823ee073e3863fb5a38ecbf1146073ff5
Summary: This change let Eden cli can ```clone``` and ```info``` on a RE Digest backed store
Reviewed By: chadaustin
Differential Revision: D28855458
fbshipit-source-id: 5582992acc5b3b3acb05b0b53d59a6768cc02491
Summary: Currently the tests manually run fsck command (Python) on the snapshot. This makes the change so that the tests will not only test the manual fsck command but also test the auto fsck (C++) so that we cover both Eden CLI fsck (Python) and Eden mount auto fsck (C++).
Reviewed By: chadaustin
Differential Revision: D29690188
fbshipit-source-id: 593db1db587c3294aad1314ea8da1d8e778df8ee
Summary:
This test is broken due to Rust panic from hg (likely there was some change in hg and the original snapshot was from 2018). It's hard to know exactly what's causing this issue because 1) it's not from eden 2) this has been broken for a while so it's tricky to bisect.
To bring back the coverage and run on a modern repo, this diff:
* generates a new snapshot (`buck run //eden/integration/snapshot:gen_snapshot -- basic`).
* adds a step to translate hgrc because the path at snapshot generation time is in it.
* migrates this test to the new snapshot
Reviewed By: kmancini
Differential Revision: D29670241
fbshipit-source-id: 1c5dccc673d516334de83582b32e2e3c9dc308f1
Summary: The hybrid changelog relies on edenapi which isn't currently supported by tests. Disable the migration for integration tests, until test repos are able to use edenapi.
Reviewed By: singhsrb
Differential Revision: D29602284
fbshipit-source-id: 8a2b4395fc5717c3880d7b74c45a0aef571cdc17
Summary: There are a lot of places in user visible text such as command line help where EdenFS is mentioned as eden/Eden/edenfs/EdenFS. This will make it consistent to 'EdenFS' in most cases. In the places where it is referring to the process/daemon, 'edenfs' will be used.
Reviewed By: chadaustin
Differential Revision: D29419151
fbshipit-source-id: 7b8296f0a0c84fdcb566ff811f7fcedbe7079189
Summary:
Like in many of the other cases, this needs to be told where the Eden binary
is instead of assuming it's right next to the edenfsctl binary, because on Buck
2 it's not.
Reviewed By: xavierd
Differential Revision: D29265845
fbshipit-source-id: 756bd863dc7d18eaf25a9ee209a9fd59345e6b5d
Summary:
This assumes the Eden binary is next to edenfsctl. Like in the previous diffs,
it isn't.
Reviewed By: xavierd
Differential Revision: D29200149
fbshipit-source-id: a7d335c5f3748c795b20480e26b24b9287708c50
Summary:
Like it says in the title. This fails because it assumes it's next to the
EdenFS binary but that's not true in Buck v2.
Reviewed By: xavierd
Differential Revision: D29200150
fbshipit-source-id: 1be5b6c647047a50dd553031691e9bbec5f9caaf
Summary: Pretty much the same as D29061439 (8f263485d9), but in another place that needs it.
Reviewed By: xavierd
Differential Revision: D29197979
fbshipit-source-id: eafccefee54a64131eb7547ad19ab1da708852e6
Summary:
On case insensitive mounts, updating between commits that have a different
casing for a file/directory would lead to the update failing due to EdenFS
believing that an untracked file is present in the mount. That conflict is
however bogus and EdenFS simply gets confused in
TreeInode::processCheckoutEntry about the entry with the different casing.
To fix this, we should avoid comparing paths in a case sensitive manner and
instead compare then in a case insensitive fashion. This allows the rest of the
checkout code to update the directory/file in place.
On Windows, there is one more subtlety: we can't change the casing of a
file/directory that is already a placeholder and thus we need to force the
entire hierarchy to be removed, this will also make the checkout fail in case
of untracked files in the hierarchy, which is also the behavior on case
sensitive systems.
Reviewed By: genevievehelsel
Differential Revision: D29121741
fbshipit-source-id: 3d2cdacf296a3d061fc828cd6d04d249542cb63f
Summary:
Similar to earlier diffs in this stack, let's not use
I think we need a solution for this in Buck 2 itself instead of within the Eden tests since this stack is getting a bit out of control, but this is probably worth doing anyway since source repo access isn't desirable (nor available) on e.g. RE.
Reviewed By: xavierd
Differential Revision: D29063216
fbshipit-source-id: 278f718b861adbf3aab9e019baaec8d84bac1433
Summary:
When running tests, we need a way for Eden to find its privhelper. Normally, it
finds it by looking next to the EdenFS binary itself, but while that works in
Buck 1, it does not work in Buck 2, where the binaries are in separate
directories
Compared to the previous diffs, this requires some extra care because on a real
system, EdenFS is running as a setuid binary, and we don't want to let people
just run whatever binary they want the EdenFS privhelper!
To that end, we reject this argument entirely if the binary is setuid.
Reviewed By: xavierd
Differential Revision: D29061439
fbshipit-source-id: bf9427211d4209cf0bea805b9ea3a53270ec455b
Summary: I'd like to use this to cache a `Optional[str]` as well.
Reviewed By: xavierd
Differential Revision: D29132026
fbshipit-source-id: 5989ae636feca061441b2c986343c8ef9759fd69
Summary:
Acessing files directory from the repo does not work well in Buck v2, because
the tests run from fbsource instead of fbcode (this also wouldn't work on RE
since the files wouldn't be there at all, but given this is to start a systemd
service that probably wouldn't work there either).
Reviewed By: xavierd
Differential Revision: D29059060
fbshipit-source-id: 09251541560cfd5879728166552ef894ba78d83c
Summary:
This is trying to access the repo directly. That works on Buck v1
because it runs from the fbcode directory. It doesn't work in Buck v2
because it runs from fbsource.
Reviewed By: xavierd
Differential Revision: D29033508
fbshipit-source-id: 504af8c205e36d98e875d13d73d15857f3a0b6a1
Summary:
Recent versions of Mercurial have a little race condition during copy
clones where the file may still be held open when it tries to be overwritten. We
don't really use this clone mechanism in production, so let's just have the test
not use it either.
Reviewed By: chadaustin
Differential Revision: D29076195
fbshipit-source-id: c19b5dc0b870653737a242200772c179549f9220
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally, we generally should not be using these singletons
in our production code.
A few final write calls need to be threaded as well.
Reviewed By: xavierd
Differential Revision: D28844011
fbshipit-source-id: 214853eea7fa65263fe6415e1ae8b76ca21512d6
Summary:
Previously if you `sudo umount -f fbsource-nfs` then try to
`eden mount fbsource-nfs`, the mount will fail because the EdenMount already
exists and is still running.
Let's properly unmount our selfs on a force unmount like we do for fuse.
There are two potential ways to detect a fource unmount: the UMNT call to the
mount deamon or the socket to the nfsd closing. The UMNT call is unreliable
(on Linux we do not get the UNMT call on `umount -l`), so this diff pursues the
socket closing option.
When the nfsd socket is closed we trigger the EdenMount unmounting process if
this has not already started.
Reviewed By: xavierd
Differential Revision: D28329482
fbshipit-source-id: 5df8f3eb818a92536095195f1b3a9e412394fbf6
Summary:
Some tests would clone using FUSE, which would thus not tests NFS. Let's make
the EdenFS class aware of NFS and force cloning with --nfs in that case.
Reviewed By: kmancini
Differential Revision: D28491492
fbshipit-source-id: ce7d609f187c21937626106a16a1f352ba88ceca
Summary:
With NFS, the server is stateless, and thus to support unlinking opened files,
the unlinked file is renamed to a "silly name". For that reason, EdenFS will
see a .nfsXXX file being created on disk.
Reviewed By: chadaustin
Differential Revision: D28484877
fbshipit-source-id: 0f4b82f97102e6efaf8899f41f644e0f47d96b1d
Summary:
For non-UTF8 names, the PathComponent constructor would raise an exception, and
since that exception wasn't caught by the handler itself, it would bubble up to
the RPC server and a generic "server IO error" would be sent back to the
client. Since non-UTF8 names aren't a server error, but an invalid argument, we
should instead return a different error.
Unfortunately, EILSEQ isn't an error that an NFS server can return, instead
let's use EINVAL as the argument is clearly invalid.
Reviewed By: chadaustin
Differential Revision: D28482032
fbshipit-source-id: b59044f1a76f7eac79e2df07356a0aeafa22e3c5
Summary: The test is passing, let's enable it.
Reviewed By: genevievehelsel
Differential Revision: D28482029
fbshipit-source-id: d8efe5fb996deb02d33a38f992e4cb18cbe97f4d
Summary:
The modeToNfsMode simply didn't consider all the mode bits to be translated to
the proper NFS mode bits. It now does.
Reviewed By: chadaustin
Differential Revision: D28459428
fbshipit-source-id: d879fb1be2085e44110ba552bc47d2770637fc86
Summary:
Similarly to the work that kmancini did for the integration tests, this
enables the Mercurial integration tests to run against NFS. Only 3 tests are
failing due to 3 separate reasons: graceful restart isn't support, permissions
given to the clients are wrong, and a lifetime issue for the Nfsd3 object.
As a bonus, I've cleaned the code a bit since TreeOnly was the only supported
configuration (and will likely stay that way).
Reviewed By: chadaustin
Differential Revision: D28459245
fbshipit-source-id: 21f043bb054d684eacaee57179f0bdf4853b0a35
Summary:
An NFS client caches the attributes of files to avoid having to request these
very frequently. What this means is that a file changed by another client (or
by the server itself) may take some time to be reflected on the client, that
time depends on the attribute caching configuration of the mount point.
For EdenFS, files can changed in 2 ways:
- Either it is changed by the user via the mount point,
- Or the user runs an `hg update`
For the first one, the client will simply update its attributes appropriately,
but for the second one, the cached attributes will only be updated when the
user does opens the file, any calls to stat prior will return the old
attributes. Since EdenFS runs on the same host, we can force the attributes
caches to be discarded by simply issuing an open call on the file that changed.
Reviewed By: chadaustin
Differential Revision: D28456482
fbshipit-source-id: 91022d35a33e436c47d94403d0c139992f880cf9
Summary:
The telemetry wrapper didn't validate the SNAPSHOT header, which makes
migrating to a new format harder. Fortunately, it doesn't even need to
read the SNAPSHOT file. The dirstate file is maintained even in EdenFS
checkouts.
Reviewed By: quark-zju
Differential Revision: D28650333
fbshipit-source-id: 174cf7039adcbb28224ec528c2462e0a9232b6cd
Summary:
Similarly to the inodes and utils change, let's also split the integration
test target. This should allow Buck to run all of this concurrently while we
would run all the tests sequentially previously.
Reviewed By: fanzeyi
Differential Revision: D28524894
fbshipit-source-id: f11d30caa35d92b0b3516ebb17c5d2c16adf8446
Summary: This diff introduces some currently failing test case to demonstrate the issue of files being created when EdenFS is not running on Windows. The tests are disabled for now since they are still failing. Later diffs should fix the issue this demonstrated and we can enable these tests on Windows at then.
Reviewed By: chadaustin
Differential Revision: D25285548
fbshipit-source-id: f0738bca05cfc82e5bf7b8238d009dc59bce93ca
Summary:
There are a few tests that we don't currently run on both git and hg,
this means we need a new decorator to run the tests on nfs.
Reviewed By: xavierd
Differential Revision: D27953136
fbshipit-source-id: f6e1fab1f763c9b878315336b2cdaa529d36ffe5
Summary: This should fix the test failure introduced in D28203778 (9bee308885) where fsck tests were not skipped on Windows.
Reviewed By: genevievehelsel
Differential Revision: D28227230
fbshipit-source-id: 97ef5717fbda567b48f02a60e1906e374fdc6766
Summary:
This applies the formatting changes from black v21.4b2 to all covered
projects in fbsource. Most changes are to single line docstrings, as black
will now remove leading and trailing whitespace to match PEP8. Any other
formatting changes are likely due to files that landed without formatting,
or files that previously triggered errors in black.
Any changes to code should be AST identical. Any test failures are likely
due to bad tests, or testing against the output of pyfmt.
Reviewed By: thatch
Differential Revision: D28204910
fbshipit-source-id: 804725bcd14f763e90c5ddff1d0418117c15809a
Summary: This just makes it more obvious _where_ `--force` should be passed.
Reviewed By: genevievehelsel
Differential Revision: D28119590
fbshipit-source-id: 1fbdb4428e9b89e7b66c959f874067485a91d534
Summary:
Currently we have limited test coverage of the NFS code. Let's start running
our integration tests on NFS mounts. We already duplicate tests to run them on
both Git and Hg repos using a python decorator. We can update this decorator to
run a copy of tests on an nfs mount.
This covers most of the tests, but a few tests do not use this decorator. See next
change.
Note some tests are currently broken, so I am using the same skip list functionality
we use for windows so we use a uniform framework.
Reviewed By: xavierd
Differential Revision: D27874662
fbshipit-source-id: c7d425830b691e395b5228d0e0f797f67987b4ec
Summary:
Some test runner don't properly redirect stdout/stderr of nested processes, or
even direct writes to filedescriptors. On these, debugging a test failure is
almost impossible for EdenFS as we rely on the test output to be interleaved
with the EdenFS logs to understand what the daemon is doing.
To solve this, we can simply create a thread that redirects the output of
EdenFS to sys.std{out,err}.
Reviewed By: kmancini
Differential Revision: D27570966
fbshipit-source-id: 6a8d5229d8d5d141e6ab423f7658744b42af46e3
Summary:
Currently eden on startup prints:
```
Starting edenfs (dev build), pid 190
Opening local RocksDB store...
Opened RocksDB store in 0.073 seconds.
Could not parse config.json file: couldn't read /var/twsvcscm/local/.eden/config.json: No such file or directory
Skipping remount step.
Started edenfs (pid 190)
Logs available at /var/twsvcscm/local/.eden/logs/edenfs.log
```
Would be convenient if it also printed session id, to be able to query scuba straight away.
Reviewed By: chadaustin
Differential Revision: D27522665
fbshipit-source-id: d7d4cf6c97bc551061761f2653375f208e393498
Summary:
I see in Sandcastle logs it was 10 minutes between `eden start` command and the following command.
So I'm curious, is it eden took 10 minutes to start or Sandcastle did something else but did not log it?
This little message in log will help to understand that.
Reviewed By: kmancini
Differential Revision: D27491709
fbshipit-source-id: 796c8db5abe49b056bd81b03ea57585a761c3cb6
Summary:
This is unused, and the Thrift file mentions that it should be fine to remove
in July 2020. It's now March 2021, time to kill it.
Reviewed By: chadaustin
Differential Revision: D26852134
fbshipit-source-id: 2872185edd834f889b78802210071d16b881e14c
Summary:
For now, this simply clone a repo with NFS, and nothing else, more of the
protocol needs implementing to support reading directories, files, etc.
Reviewed By: kmancini
Differential Revision: D26266144
fbshipit-source-id: e379e12126162f41d8d166bb53652e1e501de2e9
Summary:
Previously the code would result an exception raised while handling another
exception which is a little confusing. This diff fixes that.
Reviewed By: chadaustin
Differential Revision: D27100659
fbshipit-source-id: 8c6be4df62214c8e8d778478de66f271f7b84d3c
Summary:
As Rice is replacing the overlay, introduce some integration tests
around the subtler interactions of mode bits and status.
Reviewed By: genevievehelsel
Differential Revision: D26707622
fbshipit-source-id: 429ba404c1c0655f8fff393664bc5ba9e3b0301f
Summary:
It's silly to use `eden prefetch --no-prefetch` to efficiently glob
for filenames. Introduce an `eden glob` command which resolves a glob
relative to the current working directory.
Reviewed By: genevievehelsel
Differential Revision: D25450358
fbshipit-source-id: 45d6dc870d21510e51d5662c75e80385886899fc
Summary:
The world has moved on utf-8 as the default encoding for files and data, but
EdenFS still accepts non utf-8 filenames to be written to it. In fact, most of
the time when a non utf-8 file is written to the working copy, and even though
EdenFS handles it properly, Mercurial ends up freaking out and crash. In all of
these cases, non-utf8 files were not intentional, and thus refusing to create
them wouldn't be a loss of functionality.
Note that this diff makes the asumption that Mercurial's manifest only accept
utf8 path, and thus we only have to protect against files being created in the
working copy that aren't utf8.
The unfortunate part of this diff is that it makes importing trees a bit more
expensive as testing that a path is utf8 valid is not free.
Reviewed By: chadaustin
Differential Revision: D25442975
fbshipit-source-id: 89341a004272736a61639751da43c2e9c673d5b3
Summary:
Timeseries is memory intensive and not really required in the current context
it is being used.
Reviewed By: chadaustin
Differential Revision: D26315632
fbshipit-source-id: ee51c3ad8bef6fce152aa787c8c4602f0b499f92
Summary:
Previously, for configs that are only read once, EdenFS would have to be
stopped, the config written, and then EdenFS would be restarted. For Mercurial,
this increases the test time significantly as starting EdenFS takes ~20s.
Reviewed By: fanzeyi
Differential Revision: D26258174
fbshipit-source-id: a74d1e5be35044e95e5a7403f1bf28d557b613d2
Summary: This enables the repository to be mounted via NFS, and not FUSE.
Reviewed By: chadaustin
Differential Revision: D26229827
fbshipit-source-id: 5af5a47ebe5f1dd54df7707bf57d9b7476921f29
Summary:
The `assert_glob()` function in the glob tests previously only checked that
the expected number of results were returned. This updates the tests to
actually verify the returned paths. This also adds a few checks that verify
the behavior when the `wantDtype` parameter is True
Reviewed By: chadaustin
Differential Revision: D18903993
fbshipit-source-id: 5ed4c88160ffbda2e5fdcf089e210fda92868d56
Summary: **expected_sha1** was declared but not used in **xattr_test.py** file, removing this unused variable and variables which was used to creat **expected_sha1**
Reviewed By: xavierd
Differential Revision: D25899401
fbshipit-source-id: 26f0bb06d2c96e7f6754a4b821ffe4cf59a2f35e
Summary:
On Windows, since Mercurial doesn't yet build with Buck, we need to test
against the system Mercurial, thus remove the dependencies to //eden/scm:hg for
the tests. Also remove various dependencies that don't build yet on Windows.
This allows for the tests to run, but fail while trying to execute edenfsctl.par.
Reviewed By: kmancini
Differential Revision: D25807727
fbshipit-source-id: c2533eedc361cc6db9fdf2190476c3d52833139d
Summary:
Mercurial support files with `\` in their name, which can't be represented on
Windows due to `\` being the path separator. Currently, EdenFS will throw
errors at the user when such file are encountered, let's simply warn, and
continue.
Reviewed By: chadaustin
Differential Revision: D25430523
fbshipit-source-id: 4167b4cd81380226aead8e4f4850a7738087fd95
Summary:
Debug output belongs to stderr.
This makes it possible to turn on debug output without breaking programs
parsing stdout.
Reviewed By: singhsrb
Differential Revision: D25315954
fbshipit-source-id: c7813a824fbf6640cb5b80b5ed2d947e7059d53e
Summary:
From wez's description (slightly edited):
a symlink direction resolves outside the repo, and the new validation logic doesn't like that. the test is doing a redirect add bind then redirect add symlink. The second add is intended to implicitly delete the first one if the config is different, but it's not smart enough to realize that that is happening.
This adds an explicit `redirect del` before we do our second `redirect add`
Reviewed By: xavierd
Differential Revision: D25405011
fbshipit-source-id: 107fd272bbe830d8b23e437286ced00460902d91
Summary:
Stop implementing the legacy glob() Thrift function, since it's
deprecated, and just noise at this point.
Reviewed By: xavierd
Differential Revision: D25247641
fbshipit-source-id: a022fee169ad54c886d8f300b57bef233453fe8b
Summary:
We used to produce a confusing error message during glob evaluation
when . or .. was specified as a glob component. Instead, fail early,
with an error message that more directly explains the problem.
Reviewed By: genevievehelsel
Differential Revision: D24969096
fbshipit-source-id: fe70a8f4db1fdce8eec13890d20913b63a516518
Summary:
Be more specific about which PathComponent string failed to validate
in order to help diagnose downstream issues like glob syntax errors.
Reviewed By: genevievehelsel
Differential Revision: D24966004
fbshipit-source-id: cd3bc0aeaeb389caa13c86b91149d48c5afdb306
Summary:
I'm not entirely sure why these started failing, but enabling ui.allowmerge
made these run again.
Reviewed By: chadaustin
Differential Revision: D24697462
fbshipit-source-id: ec5ca987e7116edb12658eb7b4d03f1cf0f876d3
Summary:
Replace the old implementation of debugInodeStatus with the more
general traverseObservedInodes functionality, and add the ability to
customize its results with flags.
Reviewed By: xavierd
Differential Revision: D24300122
fbshipit-source-id: 0fbd3aa02575faa515fd7852441547d7de13426d
Summary:
We want to be able to fetch prefetch profiles on pull. That means we will need
to prefetch the contents of prefetch profiles for commits that we are not
currently on. Thus globFiles (the thrift endpoint used for prefetch profiles
fetching) needs to be able to take commit hashes to match and fetch against.
Why fetch prefetch profiles on pull? This would get the prefetch started earlier so
the files are hopefully fetched by the time the user needs them.
Reviewed By: chadaustin, genevievehelsel
Differential Revision: D23858659
fbshipit-source-id: 123e423d5117274b92405dbb5c2df690298a1c18
Summary:
On Windows, computing the sha1 of a materialized file requires opening up the
file in the working copy, as the file is cached there. Interestingly, this
potentially means that for computing the sha1 of a file, EdenFS may receive a
callback from ProjectedFS about that file or a parent directory. At this point,
EdenFS just refuses to serve this callback, as doing so may trigger an infinite
loop, or simply deadlocks. While this may sound weird, recursive callbacks are
not expected, as this signify that EdenFS view of the working copy doesn't
match what it actually is.
To close the loop, and from a code perspective, this means that computing the
sha1 of a file can fail and can throw an exception. Unfortunately, the code
didn't reflect this fact and exceptions were simply ignored, when that happens
during a checkout operation, this can leave the working copy in a weird state,
further agravating the mismatch between EdenFS view of the working copy, and
what it actually is.
Reviewed By: wez
Differential Revision: D24282048
fbshipit-source-id: 745af03189fe345150f0b1792ee1b37a1b8fb0d4
Summary:
While on Linux these can't fail (or, to be more precise: it doesnt' matter),
they can on Windows. One such exemple is when a user lock a file and triggers
an update that modifies this file. The invalidation will fail, and thus the
update should keep track of that file not being updated properly.
Previously, the invalidation would raise an exception, but that proved to be
the wrong approach as some state would need to be rolled back which the
exception didn't help in. For that, let's just return a Try and make sure that
we handle all the cases properly.
Reviewed By: chadaustin
Differential Revision: D24163672
fbshipit-source-id: ac881984138eefa65c053478a160e2a653fd3fdf
Summary:
I observed an integration test fail because Mercurial aborted with an
error about not handling EINTR, but it had no traceback. Always run
Mercurial with --traceback to try to debug these.
Reviewed By: xavierd
Differential Revision: D24204308
fbshipit-source-id: 44960bc645e5f97f61761e511b372328430fcda7
Summary:
Formats a subset of opted-in Python files in fbsource.
Black formatting was applied first, which is guaranteed
safe as the AST will not have changed during formatting.
Pyfmt was then run, which also includes import sorting.
The changes from isort were manually reviewed, and
some potentially dangerous changes were reverted,
and the directive was added to those
files. A final run of pyfmt shows no more changes to
be applied.
Reviewed By: zertosh
Differential Revision: D24101830
fbshipit-source-id: 0f2616873117a821dbc6cfb6d8e4f64f4420312b
Summary:
Now that `hg whereami` properly reads the SNAPSHOT file on Windows, the doctor
tests properly detect that Mercurial and EdenFS disagree about the current
commit, thus we can enable the remaining 2 tests.
Reviewed By: genevievehelsel
Differential Revision: D23819924
fbshipit-source-id: 21be19aff913e5e485d72e8cd730e6851ecaba2e
Summary:
on macOS we cannot safely use `fork`.
This commit replaces the use of `fork` in the startup logger subsystem.
This was a little tricky to untangle; originally (prior to any of
the `fork` removal efforts in this diff stack), the startup flow was
to spawn a set of processes via fork:
```
edenfs (setuid)
\-----edenfs (privhelper, as root)
\------edenfs (daemonized)
```
The forked children take advantage of being able to implicitly pass state to
the child processes from the parent. That data flow needs to become explicit
when removing the fork which makes some things a little awkward.
With fork removed:
* `edenfs` unconditionally spawns `edenfs_privhelper` while it has
root privs and before most of the process has been initialized.
* That same `edenfs` process will then spawn a child `edenfs`
process which starts from scratch, but that which needs to
run as the real server instance
* The original `edenfs` instance needs to linger for a while
to remain connected to the controlling tty to pass back the
startup state to the user, before terminating.
This commit deletes the check that `edenfs` is started originally
as root; previously the logic relied on the forked startup logger
continuing past the `daemonizeIfRequested` call and simply deferring
the check until after folly::init. With these changes we can't
easily perform such a check without adding some extra gymnastics
to pass the state around; the place where that is checked is in
the spawned child of the original edenfs, which is not a privileged
process and doesn't know the original euid. I don't believe this
to be a great loss as we tuck `edenfs` away under the libexec dir.
Reviewed By: chadaustin
Differential Revision: D23696569
fbshipit-source-id: 55b95daf022601a4699274d696af419f0a11f6f2
Summary: This diff fixes the eden redirection tests so it runs on Windows.
Reviewed By: xavierd
Differential Revision: D22958766
fbshipit-source-id: 45d26587831ed74d6bd7912b22c7c955b077f571
Summary:
tpx doesn't support heavyweight tags or rate limiting, and integration
tests regularly fail with timeouts on my devbig, so bump the process
start and process stop timeouts.
Reviewed By: genevievehelsel
Differential Revision: D23553924
fbshipit-source-id: fa9b8710395d61b087963d18718137e4525ae03d
Summary:
30 seconds is not enough time on heavily contented systems, including
CI. Bump the shutdown timeout to 120.
Also, correctly send SIGKILL to the daemon process when it's been
started with sudo.
Reviewed By: simpkins
Differential Revision: D22422784
fbshipit-source-id: dc7be0962705f1feb9643990309f570e352b68a0
Summary:
This test can't be non-flaky, because it relies on the kernel deciding
when to drop inodes from cache, and we've investigated it multiple
times. Given it tests a rarely-used function that would be better
expressed as a unit test in C++, just remove it for now.
Reviewed By: wez
Differential Revision: D23665455
fbshipit-source-id: 522e47113857eff399be4f2bb60e26e801d61e9a
Summary:
We intend to rename the edenfs device type to include a colon (and
possibly the backing repo basename). In preparation, update code that
detects edenfs mounts to include anything that starts with "edenfs:".
Reviewed By: genevievehelsel
Differential Revision: D23520008
fbshipit-source-id: 280f7617d5c96e23d548041b3482bca388076a7b
Summary:
We are trying to get eden running in our atypical EDA corp environment.
When testing it out on one of these machines, we got things sorted out to the point where the only test failures were coming from this file.
chadaustin identified this as being a test of dead code, and so we decided to go for a deletion of it.
If this work resumes, these tests can be retrieved from version control and then be made to work on Centos7 (hopefully at that point we'll also have contbuild/utd magic set-up to have that re-enabling automatically trigger the build/test).
Reviewed By: genevievehelsel
Differential Revision: D23463831
fbshipit-source-id: 7714547c04573b94dbb2d9acf7906734d853c5aa
Summary:
Most of the fixes are pretty trivial as the code was using functions not
present on Windows, either work around them, or switch to ones that are
multi-platform.
Of note, it looks like `hg doctor` doesn't properly detect when Mercurial and
EdenFS are out of sync, disabling the tests until we figure out why.
Reviewed By: genevievehelsel, fanzeyi
Differential Revision: D23409708
fbshipit-source-id: 3314c197d43364dda13891a6874caab4c29e76ca
Summary:
From a quick experiment, this greatly cuts down on the amount requests to
nonexistent files. For instance, the .hg directory in folders is now only
looked up once and no longer afterwards.
Reviewed By: wez
Differential Revision: D23112343
fbshipit-source-id: 223134ca591054ae9ac2e839033bbd1b714443da
Summary:
On Windows, unmounting the repo isn't sufficient to remove all traces of the
repo as ProjectedFS uses it as a cache, and thus all the files that were
touched by the user are still present on disk after unmounting it.
Reviewed By: wez
Differential Revision: D22915349
fbshipit-source-id: 445379fef57160b5da39a298ca4518e2662d16e3
Summary:
On Windows, this has the benefit of automatically converting \r\n into \n,
which allows more tests to pass.
Reviewed By: chadaustin
Differential Revision: D22871408
fbshipit-source-id: 02ec1d21dc236175c3b0f3176db9b8c91dee21a4
Summary:
On Windows, paths are separated by \, but the test was comparing them against
/. We can simply ask Mercurial to return / with the slashpath template filter.
Reviewed By: chadaustin
Differential Revision: D22871407
fbshipit-source-id: 421bd14f752f29265b12eb25609d4f65e593dda8
Summary:
Cache invalidation is hard, and on Windows we avoided doing a lot of them. It
turns out, this was the wrong decision as it's fairly easy to find cases where
the filesystem view is different from the manifest state.
Since the Linux code is most likely correct in where the invalidation is done,
let's also do the same on Windows, removing a whole lot of #ifdef. It is very
likely that as a result of this diff we end up invalidating more than needed,
thus slowing down EdenFS, but at this point I'd prefer to err on the side of
correctness, performance will come later.
While invalidating files should use PrjDeleteFile, for directories, we simply
need to mark them as placeholder, as directories created by a user won't have a
placeholder, thus ProjectedFS would bypass EdenFS when listing in.
Reviewed By: chadaustin
Differential Revision: D22833202
fbshipit-source-id: d807557f5e44279c49ab701b7a797253ef1f0717
Summary: The test is passing, enable it.
Reviewed By: genevievehelsel
Differential Revision: D22798421
fbshipit-source-id: aec5302aad38d3413385bf5f0242800d685fb5ef
Summary: The test is passing, enable it.
Reviewed By: genevievehelsel
Differential Revision: D22798424
fbshipit-source-id: 76b99457aacf5a81c2b9b3ebaedd0e6e1cf2a1e8
Summary: The test is passing, enable it.
Reviewed By: genevievehelsel
Differential Revision: D22798422
fbshipit-source-id: 0f5a106be51e319a0d317900cc88de1131b95e4d
Summary: Out of the 5 tests, 3 are passing, let's enable them.
Reviewed By: genevievehelsel
Differential Revision: D22798418
fbshipit-source-id: 5bd3fd90945e5556de7838a5dc61ad00865a6d55
Summary: Half of them are passing, let's make sure they run.
Reviewed By: genevievehelsel
Differential Revision: D22798423
fbshipit-source-id: b762213ffad431de6f54acb8ceffa54f28f5909d
Summary: The test is passing, enable it.
Reviewed By: genevievehelsel
Differential Revision: D22798417
fbshipit-source-id: c45531a75a3db61e33f95ef80eb007f12dd02f2a
Summary: The test is passing, enable it.
Reviewed By: genevievehelsel
Differential Revision: D22794679
fbshipit-source-id: 8d5f5a9b9fd7750034baeda814bfeff29882d409
Summary: These are all passing, enable them.
Reviewed By: genevievehelsel
Differential Revision: D22794158
fbshipit-source-id: c381a906e096c3642248a2521e8e7772d74f992e
Summary: All of the tests are passing, enable them.
Reviewed By: genevievehelsel
Differential Revision: D22794159
fbshipit-source-id: 8b7e33f3abbde88e06488f7fe9ea7217d204e70e
Summary: The test is passing, enable it.
Reviewed By: chadaustin
Differential Revision: D22794161
fbshipit-source-id: 94d167b88782d386ca60b9215f1c2aef56a88a4d
Summary: The only ones not passing are the FUSE ones, which is expected. Enable all the others.
Reviewed By: genevievehelsel
Differential Revision: D22794162
fbshipit-source-id: 31f51fd5fff8e3ced75924533fa536208dabf11d
Summary: The test is passing, enable it.
Reviewed By: chadaustin
Differential Revision: D22794155
fbshipit-source-id: 286c9f1129d279487019206d58521951c768cbb1
Summary: 2 out of 3 tests are passing, let's run these.
Reviewed By: chadaustin
Differential Revision: D22794160
fbshipit-source-id: e62e1956980689b083107ebb3b8340880e0d72a6
Summary:
Besides the 3 listed, all the other 13 are passing, let's make sure we run them
to not regress.
Reviewed By: chadaustin
Differential Revision: D22794164
fbshipit-source-id: 5660cc36365de96a2b9e62e3462c01b39904d2f3
Summary:
we've seen flaky tests here on stress test runs, lets subtract time.time() by an epsilon of one to ensure that the comparison is only made within a second precision (time.time() worst case precision)
an example error: `AssertionError: 1595414015.5022793 not greater than or equal to 1595414015.5025299`
`AssertionError: 1595759609.439305 not greater than or equal to 1595759609.441682`
Reviewed By: chadaustin
Differential Revision: D22687496
fbshipit-source-id: 8e87148d620577e3198d2845d785a87a909cd1d3
Summary:
Disregard two default systemd services on Fedora 32 that cause our
systemd tests to fail.
I believe genevievehelsel is planning on removing this code soon anyway.
Reviewed By: genevievehelsel
Differential Revision: D22713393
fbshipit-source-id: b703b23a3158cb007dc2e1cb53fae36be7282719
Summary:
We are unifying C++ APIs for accessing optional and unqualified fields:
https://fb.workplace.com/groups/1730279463893632/permalink/2541675446087359/.
This diff migrates code from accessing data members generated from unqualified
Thrift fields directly to the `field_ref` API, i.e. replacing
```
thrift_obj.field
```
with
```
*thrift_obj.field_ref()
```
The `_ref` suffixes will be removed in the future once data members are private
and names can be reclaimed.
The output of this codemod has been reviewed in D20039637.
The new API is documented in
https://our.intern.facebook.com/intern/wiki/Thrift/FieldAccess/.
drop-conflicts
Reviewed By: yfeldblum
Differential Revision: D22631599
fbshipit-source-id: 9bfcaeb636f34a32fd871c7cd6a2db4a7ace30bf
Summary:
Bypass truncation-based transaction if narrow-heads is on.
The transaction abort still works logically because commit references stay
unchanged on abort.
Related EdenFS and Mononoke tests are updated. Mononoke tests probably
shouldn't rely on revlog / fncache implementation details in hg.
Reviewed By: DurhamG
Differential Revision: D22240186
fbshipit-source-id: f97efd60855467b52c9fb83e7c794ded269e9617
Summary:
An assertion error is raised if `eden doctor` is in the middle of a merge. This is because we enter a specific "if" condition in the case that mercurial has two parent commits, and EdenFS only ever tracks `p0`, so EdenFS simply sets `p1` to the null commit in `_select_new_parents()`. Specifically, this is in the case in which both `_old_dirstate_parents` and `_old_snapshot` are not None.
Because `_old_dirstate_parents` has `p1` set to nonnull, and Eden thinks it is null , the check `self._new_parents != self._old_dirstate_parents` would be `True` even though there was actually no error.
Reviewed By: chadaustin
Differential Revision: D22048525
fbshipit-source-id: 9a19cc092e2bd80db0e01fb38533a1007640bee6
Summary:
On Windows, paths are case insensitive (but the filesystem is case preserving),
and thus `open("FILE.TXT")` and `open("file.txt")` refer to the same file. When
that file is not materialized and its parent directory isn't yet enumerated,
PrjFS will call the PRJ_GET_PLACEHOLDER_INFO_CB with the file name passed in to
the `open` call. In this callback, if the passed in name refers to a valid
file, it needs to call PrjWritePlaceholderInfo to populate the directory entry.
Here is what the documentation for that function states:
"For example, if the PRJ_GET_PLACEHOLDER_INFO_CB callback specifies
dir1\dir1\FILE.TXT in callbackData->FilePathName, and the provider's backing
store contains a file called File.txt in the dir1\dir2 directory, and
PrjFileNameCompare returns 0 when comparing the names FILE.TXT and
File.txt, then the provider specifies dir1\dir2\File.txt as the value of
this parameter."
While the documentation doesn't state how that name is used internally, we can
infer (and test) that the returned case will be used as the canonical
representation of that file, ie: the one that a directory listing will see.
Since the PathMap code already does a case insensitive search, we just need to
make sure to use what it returns instead of re-using the name used for the search.
The only caveat to all of this is the original comment that describe that
`metadata.name` can't be used as it causes crashes. From what I can tell, this
was written in later 2018, and I believe is no longer relevant: the
`metadata.name` field was simply not populated.
Reviewed By: wez
Differential Revision: D21799627
fbshipit-source-id: aee877cc2d5f057944fcd39b1d59f0e97de6315c
Summary:
updated `eden top` to:
- obtain PID-fetchCounts data from the updated -`getAccessCounts` thrift call in the previous diff
- display that data in a new column `FUSE FETCH`
Reviewed By: kmancini
Differential Revision: D22101430
fbshipit-source-id: 6584e71ce3a4629c73469607ca0a4c6ffd63e46f
Summary: The word "disabled" better express the intent behind it.
Reviewed By: fanzeyi
Differential Revision: D21945736
fbshipit-source-id: a61a3df4f1b704224e7f4d4a7f1a43f4d2b5b2aa
Summary:
Removing a directory with unlink raises different errors on Windows: EACCES
instead of EISDIR.
Reviewed By: fanzeyi
Differential Revision: D21944455
fbshipit-source-id: 251ceb305a46c01ece78f45b0fa8000778c6c650
