Summary: Don't set the "fb" cargo feature via cmake unless we are in fb build.
Reviewed By: quark-zju
Differential Revision: D33781554
fbshipit-source-id: 39c61dc1631ebfd8a2c858b5e214c6a09ce0b005
Summary:
The current serialization type we use for the graceful restart protocol only
allows us to send mount specific information in the non error case.
If we want to send general takeover information, like the order of the file
descriptors we send along with the takeover message, we are out of luck.
I am changing the final type we send during takeover to be a struct so that
we can add non-mount specific information. This allows us to create an explicit
order of our file descriptors and will let us send a dynamic number of general
file descriptors.
Currently, we have to send all the expected general file descriptors
(thift socket, lock file, mountd socket) in a fixed order.
And those file descriptors have to be valid because we cannot send an invalid
file descriptor with sendmsg.
When NFS is not enabled we do not initialize the mountd socket so we cannot
send a valid filedescriptor for the mountd socket (unless we sent some dummy
file descriptor which feels really hacky and not a good permanent solution).
To be able to fix the bug we have in graceful restart due to not having
a mountd socket we need to be able to send only a portion of the general
file descriptors.
When NFS is not enabled, we do not initialize the NFS Server. This causes
a seg fault during takeover because we try to shutdown the nfs server.
We should only shutdown the NFS Server when we initialized it. This
means we only pass the mountd server socket during takeover when NFS was enabled
when we started the server.
To allow use to enable the NFS Server across graceful restart we only
perform a takeover start when the mountd server socket was received during
takeover.
Reviewed By: xavierd
Differential Revision: D33755447
fbshipit-source-id: cc471c3051de8e41c21a4d1d8e42d93d5295e90e
Summary:
In one of my next changes I am going to be introducing a integration test where
NFS is not enabled.
Previously, we did not have any tests that ran with NFS disabled and so a bug
slipped through the cracks in graceful restart when NFS is disabled.
If I am going to disable NFS then these tests should not pretend to run on NFS.
This introduces a parameter to the eden test decorator that allows skipping
the NFS tests.
Reviewed By: genevievehelsel
Differential Revision: D33247961
fbshipit-source-id: ab37bd1aa64b9b09a69fe5e90e8068d4f444cd29
Summary:
When an update removes a directory but a materialized ignored file/directory
was present, EdenFS will rightfully not remove that ignored file/directory.
However, the parent directory would have been a placeholder on disk. And since
placeholder directories can never be made full[0], the directory keep its
placeholder. Listing this directory would thus issue an opendir callback, but
EdenFS would fail due to that directory not being present in the manifest.
Since this case can only happen in this case, let's simply return an empty
directory and let ProjectedFS merge it with the on-disk directory content.
[0]: https://docs.microsoft.com/en-us/windows/win32/projfs/cache-state
Reviewed By: genevievehelsel
Differential Revision: D33676621
fbshipit-source-id: bdbf769175a6471a1a709bdb0b03a45bd60ef366
Summary:
Now that notifications are running in a serial executor, and are also issuing
disk IO, notifications are significantly slower than they used to be. While
writing to the working copy is an overall anti-pattern, some workflows (like
Unity) do and it's thus critical that their performance isn't affected
negatively.
In order to solve this, we can simply move the handling of notifications to the
background and answer the notification immediately, since notifications can no
longer fail, we shouldn't need to send an error back to ProjectedFS which would
anyway be ignored.
The drawback is of course that applications are no longer blocked while the
notification is being processed in EdenFS, and thus any operation that needs to
get a synced up inode hierarchy will need to wait on all the pending
notifications.
Reviewed By: chadaustin, genevievehelsel
Differential Revision: D32480993
fbshipit-source-id: 7ad6b07f540f7d9a52a35a0ff3b94911ef5267af
Summary:
With Facebook having been renamed Meta Platforms, we need to change the license
headers.
Reviewed By: fanzeyi
Differential Revision: D33407812
fbshipit-source-id: b11bfbbf13a48873f0cea75f212cc7b07a68fb2e
Summary:
It feels a bit weird for getDaemonInfo().status's meaning to diverge
from fb303's getStatus(), but fb303 is being restructured so remove
the custom getStatus implementation.
Reviewed By: praihan
Differential Revision: D33176576
fbshipit-source-id: 5a7b7f3394b123b1073191508c2c0c1ebcda5bb5
Summary:
The EdenApi methods requiring `reponame`. It leaks the HTTP URL detail to many
more layers.
This diff makes the `Client` use the repo name provided by the config, or
explicitly via the `repo_name` API (as in tests). The HTTP client uses the repo
name in its config so callsites don't need to provide the name again.
Previously, if reponame is not provided, the HTTP client builder will use
"unknown" as fallback. That behavior is changed to an error instead.
The above change breaks some tests using production configs on scmstore
construction. scmstore is updated to properly avoid constructing EdenApi
if repo name or URL is not provided, instead of constructing an EdenApi
HTTP client using the wrong "unknown" URL silently.
This diff does not change the `EdenApi` trait yet, since that is a large change.
Reviewed By: yancouto
Differential Revision: D32800595
fbshipit-source-id: a156e18b2f6fc41804cc40e3c926086be63fd9b9
Summary:
This option is only ever used for tests, let's remove it and simplify the code
a bit.
Reviewed By: singhsrb
Differential Revision: D33103247
fbshipit-source-id: e822de6dba541152452984f6aad5258a5585db02
Summary:
From the Thrift documentation:
* Ensure that all inflight working copy modification have completed.
*
* On some platforms, EdenFS is processing working copy modifications
* callbacks from the platform in an asynchronous manner, which means that by
* the time a write/creat/mkdir/unlink/etc syscall returns from the kernel,
* EdenFS may not have updated its internal state.
*
* Thus, an application making changes to the working copy and quickly
* requesting EdenFS to perform an operation on it will race with EdenFS
* updating its internal state and may thus get stale data.
*
* To avoid this, EdenFS queries need to internally synchronize the working
* copy before performing the query itself. This structure defines how EdenFS
* will do this.
*
* Applications that care about synchronizing EdenFS up to a certain point in
* time are expected to set a non-zero syncTimeout once to synchronize EdenFS
* and then issue all their thrift requests with a syncTimeout of 0.
Reviewed By: chadaustin
Differential Revision: D32900351
fbshipit-source-id: c242d881b2f2db6cbca544288a0eac5959469fc5
Summary:
The test started failing on CI due to the file not being read when being
locked. ProjectedFS behavior when the file is a placeholder vs a full file
differs when that file is locked and invalidated. In the case of a placeholder,
the file will be removed, while the full file won't. Let's force a full file by
simply reading it before locking it.
Reviewed By: kmancini
Differential Revision: D32776736
fbshipit-source-id: faf0bca1becc4bfed8cd6bc804e33a0540f3b8cb
Summary:
There was a subtle bug in D31782916 (87ef50135e) where a wildecarded enumeration may send
the same directory entry multiple times to ProjectedFS. This can happen if the
first directory entry returned to ProjectedFS is not where iter_ initially
points to, which is almost always the case when a wildecard is being used. Thus
the first call to advanceEnumeration will simply find the first element sent to
ProjectedFS and send it again on the next readdir call.
Reviewed By: fanzeyi
Differential Revision: D32742433
fbshipit-source-id: 12904a5d2df4407072b8d775950f92ea93344a73
Summary:
Now that takeover is implemented on nfs, we can enable the integration tests
for graceful restart on NFS.
This last test relies on file invalidation, so we implement this as fully as we
can on an nfs mount in order to enable this test.
Reviewed By: xavierd
Differential Revision: D32225243
fbshipit-source-id: c6ddbba54eb71edf6bfe70285a775d438a9d0acd
Summary:
Hardlinks aren't supported, but we don't have tests verifying this. While we
may support them in the future, let's make sure we don't break them until we
do.
Reviewed By: chadaustin
Differential Revision: D32480994
fbshipit-source-id: ae00f605a3cea8ebd4fbcba06098d109c0efa8e1
Summary:
Now that graceful restart is supported, we can enable the integration tests!
These are all passing. The IO test is a little flakey, but the fuse version is
as well. So I haven't invested a lot of time in it.
One test is only compatible with version 1 and 2 of the takeover protocol
which is not compatible with takeover for nfs mounts. So that test will remain
disabled.
Reviewed By: xavierd
Differential Revision: D31738202
fbshipit-source-id: 0160310355138db59a8e671d3888b948e896eefa
Summary:
Now that we have NFS mounts we need to support takeover for those mounts.
This commit teaches the takeover server and client to serialize and deserialize
NFS information.
NFS information includes:
- the mountd socket. Mountd is used to register mount points, and should keep
the same socket across graceful restarts.
- a connected socket for each of the NFS mount points.
- All the rest of the mount point information that we send for fuse mounts (with
the exception of the fuse negotiated parameters.
Future commits will teach the NFS components to graceful stop and yield this info
as well as restart with this info.
Reviewed By: xavierd
Differential Revision: D31802787
fbshipit-source-id: 667625589b77eed79f7e17390013a818d4be7068
Summary: Instead of fetching the file size and sha1 of a file separately, we use the new FileInode::getBlobMetadata() method to get them at the same time.
Reviewed By: xavierd
Differential Revision: D32401043
fbshipit-source-id: 7fcedeaf9070931e55d3fd8b23544b8b56118d1a
Summary:
Adds logic for new thrift endpoint that allows users to request attributes for a given list of files.
Requested attributes are obtained for each file and stored in a FileAttributeData struct. Each file is then mapped to a FileAttributeData struct (or an EdenError). If a requested attribute cannot be found for a given file, an EdenError is returned instead of a FileAttributeData.
Integration tests cover too many cases, so I've commented out the excess cases and will turn them into unit tests in a later diff. For now, they are available for use, but not included by default.
Integration tests cover the following cases:
- Requesting all attributes for 2 files
- Requesting attributes for a non-existent file
- requesting just file_size for a file
Commented out Integration tests include:
- Requesting attributes for empty string path
- Requesting attributes for a file that contains dot components
- Requesting attributes for a directory
- Requesting sha1 for a symlink (should file_size for a symlink also return an EdenError?)
- requesting all attributes for no files
- requesting no attributes for a file
- requesting just sha1 for a file
Reviewed By: kmancini
Differential Revision: D32179305
fbshipit-source-id: 6c23bb7780154159fdd9626d332c641311afbe57
Summary:
In every test setup, writing a file to the repository will always call `hg add`
in order to this newly added file to be added in subsequent `hg commit`. Since
`hg add` can be very expensive (in particular on Windows), some tests are
spending most of their time calling `hg add` instead of running the body of the
test. As an easy way to avoid this, let's stage the added files in memory
first, and then call `hg add` once just before `hg commit`.
Reviewed By: genevievehelsel
Differential Revision: D32333775
fbshipit-source-id: 8cc6cfad7ec73f6f0c50d7d1e6affa6cdd828e2f
Summary:
The new sparse profile config logic was triggering during an Eden code
path, where sparse doesn't exist. This caused a crash. Let's check for sparse
before trigging the path.
This only occured if there was another exception during checkout.
Reviewed By: quark-zju
Differential Revision: D32436881
fbshipit-source-id: 6d5da888e4f8a62a05a390c7e87bbc3061356fc2
Summary:
In order to compute the sha1 of materialized files in the working copy, EdenFS
on Windows will open and read the file out of the working copy directly. In the
case where EdenFS is wrong about the state of the file, this can lead to a
ProjectedFS callback to be triggered which may need to take the same lock that
the sha1 computation holds. This obviously results in a deadlock and to protect
against this, EdenFS would bail early when detecting recursive callbacks.
With read-only callbacks having been switched to only take into account the
underlying Mercurial data, and not the inode state, these callbacks will never
attempt to take any inode locks. This also means that the deadlock described
above can no longer happen: the ObjectStore layer will never attempt to read
anything from the working copy, thus recursive callbacks are impossible. These
checks are thus unecessary.
In theory, these checks could stay in place as a safety measure, but EdenFS
will soon need to list directory entry and call PrjGetOnDiskFileState in
response to file/directory creation/removal to handle the case where these can
happen out of order.
Reviewed By: genevievehelsel
Differential Revision: D32149411
fbshipit-source-id: 73b228bb8c73416b38f7f2042c25964e1aef3c4e
Summary:
In ProjectedFS terminology, the "backing store" refers to the state of files
and directories in the source control backend. For the longest time, EdenFS
assumed that the Inodes hierarchy was the "backing store", but this lead to
subtle bugs. For instance, renaming a non-hydrated placeholder would render the
file unreadable due to the read callback from ProjecedFS callback using the old
file name, not the new one
(https://github.com/microsoft/ProjFS-Managed-API/issues/68). It also meant that
files just created would always get a tombstone written on disk when being
removed as ProjectedFS queries EdenFS when removing a file via the access
callback. Returning true from this callback means that this file is present in
the backing store and thus ProjectedFS needs to create a tombstone on disk.
In order to fix these issues, EdenFS needs not to query the Inodes hierarchy,
but the Tree hierarchy. The latter matches with the expected behavior from
ProjectedFS.
For modification callbacks, these are by design acting on the Inode hierarchy
and thus are unchanged.
Reviewed By: chadaustin
Differential Revision: D32022639
fbshipit-source-id: bef18a0709a7eff9ef0a48a90126d1339be0023e
Summary: This should give us more diff time signal.
Reviewed By: chadaustin
Differential Revision: D32207108
fbshipit-source-id: c9f6bc947a2622021194f8fa3d673cd61ae9bb2a
Summary: sys.platform is "win32", not the path to python.
Reviewed By: fanzeyi
Differential Revision: D31554106
fbshipit-source-id: 64b388d2fb8a493f811a0cf22fe2471a25bfbf7e
Summary: This appears to have broken all the tests on Linux.
Reviewed By: zhengchaol
Differential Revision: D31505082
fbshipit-source-id: 610eb941d0f0eb536a4688ac2637a8466be0b05c
Summary:
On Windows, edenfsctl.real is a par file that Windows can't execute directly.
Thus let's have Python run it.
Reviewed By: fanzeyi
Differential Revision: D31477721
fbshipit-source-id: d5a699ceb3d3b1b3d5778ef5720bca7c299bed80
Summary:
These tests have been failing for a long time due to the expected output on
Windows containing \ but the test using the / separator.
Let's simply use os.path.join to build a path with the right separator in these
tests.
Reviewed By: fanzeyi
Differential Revision: D31477722
fbshipit-source-id: a4ac25750647229974e23e305508e83917011fef
Summary:
The goal of this stack is to remove Proxy Hash type, but to achieve that we need first to address some tech debt in Eden codebase.
For the long time EdenFs had single Hash type that was used for many different use cases.
One of major uses for Hash type is identifies internal EdenFs objects such as blobs, trees, and others.
We seem to reach agreement that we need a different type for those identifiers, so we introduce separate ObjectId type in this diff to denote new identifier type and replace _some_ usage of Hash with ObjectId.
We still retain original Hash type for other use cases.
Roughly speaking, this is how this diff separates between Hash and ObjectId:
**ObjectId**:
* Everything that is stored in local store(blobs, trees, commits)
**Hash20**:
* Explicit hashes(Sha1 of the blob)
* Hg identifiers: manifest id and blob hg ig
For now, in this diff ObjectId has exactly same content as Hash, but this will change in the future diffs. Doing this way allows to keep diff size manageable, while migrating to new ObjectId right away would produce insanely large diff that would be both hard to make and review.
There are few more things that needs to be done before we can get to the meat of removing proxy hashes:
1) Replace include Hash.h with ObjectId.h where needed
2) Remove Hash type, explicitly rename rest of Hash usages to Hash20
3) Modify content of ObjectId to support new use cases
4) Modify serialized metadata and possibly other places that assume ObjectId size is fixed and equal to Hash20 size
Reviewed By: chadaustin
Differential Revision: D31316477
fbshipit-source-id: 0d5e4460a461bcaac6b9fd884517e129aeaf4baf
Summary: This diff fixes some integration test errors after enabling the new edenfsctl.
Reviewed By: xavierd
Differential Revision: D30789741
fbshipit-source-id: 02d74defc41def4fb6ea0cc4694f944b4c0044e2
Summary: This diff adds the Rust edenfsctl to our open source build.
Reviewed By: xavierd
Differential Revision: D30788685
fbshipit-source-id: 603caa933ecfc5af0ede7e22f6c7911462da3a65
Summary:
We run periodic inode unloading for unlinked inodes on NFS because we get no
information from the client on when inodes are no longer needed, and we have to
clean them up at some point for memory and disk reasons. See previous commit
summaries for more details on this (D30144901 (ffa558bf84)).
Let's add some counters on this so we have a bit more visibility into the
process. This counter is meant to mimic the PeriodicUnloadCounter counter.
Reviewed By: chadaustin
Differential Revision: D30966688
fbshipit-source-id: cfc8d769b53073d9f4c0c27b6bee20e222c6c8d2
Summary:
It can be surprising when a job suddenly is no longer able to run sudo, or no
longer run as root that all the tests are marked as being successful, despite
the fact that they no longer run. Let's recognize when we run on EdenFS to
allow tests to fail if they can no longer run EdenFS.
Reviewed By: zhengchaol
Differential Revision: D30357402
fbshipit-source-id: c3758d7a5a3c575dd68bd97062ae24abe4124874
Summary:
Now that we might have multiple kernel protocols per mount (i.e. both fuse and
nfs on macOS) let's include them in eden rage.
Reviewed By: xavierd
Differential Revision: D31154042
fbshipit-source-id: 38e7630829d70fe9dd6dbeabacc3b538ee798e0d
Summary:
We periodically need to dereference inodes on NFS because we get no other info
from the kernel on when should dereference them.
This means the NFS kernel might have references to inodes after we delete them.
An unknown inode number is not a bug on NFS. It's just stale, so the error should
reflect that.
Reviewed By: xavierd
Differential Revision: D30144898
fbshipit-source-id: 3d448e94aea5acb02908ea443bcf3adae80eb975
Summary:
`eden prefetch` and `eden glob` return lists that despite being called
"maching files" actually contains both files and directories.
In some cases, we only want the list of files and it introduces unnessecary
overhead on our clients for them to have to stat all the files in the list to
filter out the dirs. Let's add an option to just list files.
Reviewed By: chadaustin
Differential Revision: D30816193
fbshipit-source-id: 6e264142162ce03e560c969a0c0dbbc2f418d7b9
Summary: the rage summary is getting hard to quickly parse, so this underlines each section header, as well as unifies underline looks (with `eden stats`). This adopts the underline code from `eden du` and makes it a util function for shareability.
Differential Revision: D30857773
fbshipit-source-id: 66b5b06f5b0125304d45d3465a8bc2248693b791
Summary: This test can fail for not having `USERNAME` environment variable when the test is run as root. Let's just skip the test when this happens because it doesn't make sense to drop priv as root.
Reviewed By: xavierd
Differential Revision: D30868518
fbshipit-source-id: 14ff6db218b1477f5905f2df3ad075a5ca186117
Summary:
This change has the unintended effect of causing any Thrift calls to
potentially issue a recursive EdenFS call due to symlink resolution requiring
running `readlink` on the root of the repo itself.
Fixing this isn't really possible, thus let's revert the change altogether, we
can force clients to issue a realpath before issuing EdenFS Thrift calls.
Reviewed By: kmancini
Differential Revision: D30550796
fbshipit-source-id: 9494c8e08c8af2392eeb344879f156cb56f93ea6
Summary:
The rate metric can be unreliable, and in some environment is never updated by
the time it is read by the test. Since this test cares about the number of
events, using count is a better metric, and more reliable.
Differential Revision: D30377128
fbshipit-source-id: f10c656567e3a3c07b66ecc6fc563a53199e3088
Summary:
In the ASIC test environment, we've see cases where this isn't set, let's
simply skip the test in this case instead of failing.
Reviewed By: chadaustin
Differential Revision: D30353929
fbshipit-source-id: 956da6f8f12de025b8ca72e40097f1f9d50e6bf7
Summary:
In the case where the path to the mount has symlinks, EdenFS would only accept
the path to it that was specified at mount time, even though another path may
refer to the same directory.
To solve this, we can simply normalize paths in all the Thrift endpoint to make
sure that EdenFS always refers to a mount point under its non-symlinked path.
Reviewed By: chadaustin
Differential Revision: D30320515
fbshipit-source-id: e578d059a3b1307d6b24c4b9bdb1ceb3b534c460
Summary:
The changes in this diff makes the code to "run" and skip correctly according to our rules.
All integration tests are now runnable on Windows with mode/win
Differential Revision: D30143255
fbshipit-source-id: b2ddbff7268f182274b3755f4b28df6ac6cdeef4
Summary:
This newly added config controls the default behavior of `eden clone` and
whether a new clone will use NFS or FUSE. This is intended to facilitate the
transition to NFS from FUSE on macOS.
Differential Revision: D30110056
fbshipit-source-id: ea6b493aa803297952b46434f6d11d8edf58e40b
