Summary: v1 and v2 have both been written out forever, so I think we can drop the v1 format.
Reviewed By: zzl0
Differential Revision: D46075828
fbshipit-source-id: 373dae9cf6a53534489022716986aa85b5de5552
Summary:
We don't use the Cython-based Thrift client because it never worked in
the open source build. Removing it fixes the tests when run with
nix2rpm Python binaries.
Reviewed By: xavierd
Differential Revision: D45508963
fbshipit-source-id: d6f0e162177082a78504a3a6014a8a4d32670ccc
Summary: In the next diff, we'll use this util function to acquire an exclusive handle to a file for a prjfs integration test.
Reviewed By: xavierd
Differential Revision: D44349708
fbshipit-source-id: ded7037266f539c681569f0c90f875717760fed5
Summary:
hg grep under the hood is `hg st | xargs grep`. The return code is the xargs
return code. xargs on mac and linux behave a little differently.
according the the linux man page:
```
EXIT STATUS
xargs exits with the following status:
0 if it succeeds
123 if any invocation of the command exited with status 1-125
```
according to the mac man page:
```
EXIT STATUS
The xargs utility exits with a value of 0 if no error occurs. If utility cannot be found, xargs
exits with a value of 127, otherwise if utility cannot be executed, xargs exits with a value of
126. If any other error occurs, xargs exits with a value of 1.
```
So a failure on linux in grep will result in a 123 code while on mac it will
result in a 1 exit code. I wish mac behaved like linux, but it doesn't. So lets
teach the tests to accept the xargs return code based on the platform.
Reviewed By: mshroyer
Differential Revision: D43924480
fbshipit-source-id: 0e6c2ec2c6a7553a1aed7c072f0464e5d1fa64c7
Summary:
Migrate off using legacy filelog repo and bundle2 protocols to modern
protocols. This fixes the test.
Reviewed By: muirdm
Differential Revision: D43517041
fbshipit-source-id: c997bd9863b849b4410a3fc163b67d6ec6b98dbd
Summary:
Migrate off using legacy filelog repo and bundle2 protocols to modern
protocols. This fixes the test.
Reviewed By: muirdm
Differential Revision: D43516085
fbshipit-source-id: f24ac78dbeb66192d4580fd6f74fb38a833d09d4
Summary:
EdenFS assumes in several places that even materialized directories will
receive readdir requests. This assumption is slightly wrong on Windows in the
case where a directory hierarchy is completely removed, and then reconstructed
at the identical, with say a filesystem revert operation. In that case, EdenFS
will not participate in any readdir requests in this directory hierarchy as no
directories are placeholder.
During checkout, this assumption is used to dematerialize directories in
saveOverlayPostCheckout, in some cases however, this can lead to dematerialized
directories being contained in a non-placeholder directory on Windows. This
case in particular is fraught with issues. For instance,
invalidateChildrenNotMaterialized may invalidate these dematerialized
directories but doing so would make these disappear entirely due to EdenFS not
participating in non-placeholder directories! I believe a similar issue may
also be possible with future checkout operations, although the exact steps
elude me.
To solve the above, one easy way is to simply force directories traversed by
checkout to always have a placeholder added to them. This is achieved by
invalidating the directory. Since it's unclear if we could also always
invalidate the directory on Linux and macOS, let's add a config so we can roll
this out slowly.
Reviewed By: kmancini
Differential Revision: D42483525
fbshipit-source-id: 9d8c9fe3db8914ca2a46a77f10cf77f2ff871ad9
Summary:
When EdenFS crashed and/or was killed, only `hg update` would provide a
workaround to continue the checkout, `hg status` wouldn't. Let's add the
recovery steps in there too as users may run `hg status`.
Also, change the remediation for doctor to mention restarting the `hg update`
instead of always recommending recloning.
Reviewed By: kmancini
Differential Revision: D41168693
fbshipit-source-id: a960b978d7d1ec0db5619215de2eecfde16f0afc
Summary: Add basic support to sniff ".hg" vs ".sl" dot dir, and otherwise default based on env vars.
Reviewed By: quark-zju
Differential Revision: D39900481
fbshipit-source-id: d9900755f54711a9d067d8fd65f2fc25aa278a7a
Summary:
While unix sockets are rare on Windows, some tools lay them in the working
copy. EdenFS was however not recognizing them and failed to update its inodes
hierarchy which could lead to wrong `hg status` or failed `hg update`.
Reviewed By: kmancini
Differential Revision: D39227982
fbshipit-source-id: b1013c9835e0a1718defdbcf98b666ef32fb9e33
Summary: this test currently fails because `hg config extensions.eden` exits with code 1.
Reviewed By: mshroyer
Differential Revision: D38847188
fbshipit-source-id: c7783fad144fa2e82f156c8e5df66e63d30f95a0
Summary:
The use of both thrift-py and thrift-py3 forces the code to awkwardly convert
between types. Since these tests are already only compatible with Linux+Buck,
there is no downside of just using the thrift-py3 API exclusively to avoid
these conversion. This makes the code easier to read, and slightly reduces our
dependencies.
Reviewed By: mshroyer
Differential Revision: D38769915
fbshipit-source-id: 14069f051bd34df03e5f0a5cfb710fae83144143
Summary:
We're seeing random timeout in this test in between EdenFS being killed, and it
being restarted. Looking at the code, the main culprit appears to be the
unbounded subprocess.wait call which Python's documentation warns about:
https://docs.python.org/3/library/subprocess.html?highlight=subprocess#subprocess.Popen.wait:
> Note: This will deadlock when using stdout=PIPE or stderr=PIPE and the child
> process generates enough output to a pipe such that it blocks waiting for the
> OS pipe buffer to accept more data.
Since the _process object is created with subprocess.PIPE, this looks to be a
good candidate for what we're seeing.
Instead of using subprocess.wait, we can also use subprocess.poll combined with
util.poll_until for the same behavior, but without the risk of deadlock.
Reviewed By: fanzeyi
Differential Revision: D38395869
fbshipit-source-id: acbae227644d9265cd3826102fbf5309f2a404d5
Summary: My new understanding of loaded vs materialized is that materialized indicates the files are potentially different from what's on disk and loaded are files whose inodes are loaded right now. Let's update this test accordingly to ensure certain directories were not loaded.
Reviewed By: xavierd
Differential Revision: D37975865
fbshipit-source-id: c8ceb722396c8fcfe14dbac14a9d03bd63fedd17
Summary: This test was disabled due to a test failure. Turns out the Windows implementation just considers certain files as being conflicts. Doing 'hg update --clean' fix it, and still satisfies the criteria of not needing to reclone.
Reviewed By: xavierd
Differential Revision: D37936355
fbshipit-source-id: ece36afad210f4a11964969dcc7a48f4c961de53
Summary: The old way has issues with tpx, according to xavierd.
Reviewed By: xavierd
Differential Revision: D37855991
fbshipit-source-id: 579627c9840e9b14d26f9d1668e1f78e9d48d593
Summary:
Add a test for confirming that Eden can resume a checkout after it was
previously killed.
Reviewed By: xavierd
Differential Revision: D36986501
fbshipit-source-id: 66501ee6df74e92db1b0d4727b4773f8d38db2d6
Summary:
In D36914467 (e7918c1d11), a very subtle change was introduced in the processBothChanged
lambda. The path component being used before was the one from the tree, while
it is now coming from the inode itself. This change caused the
test_change_casing_with_untracked to start failing with:
AssertionError: {'DIR2/untracked': '?'} != {'dir2/untracked': '?'}
- {'DIR2/untracked': '?'}
? ^^^
+ {'dir2/untracked': '?'}
? ^^^
Or in english, the test expected the untracked file to be in the directory
DIR2, but is now in the directory dir2. Reading the test itself, and the
comment just above the failed assert makes it clear that on Windows, the dir2
directory will be on disk. It thus makes more sense for status to report the
untracked file from the dir2 directory instead of from the DIR2 directory.
Reviewed By: DurhamG
Differential Revision: D37536084
fbshipit-source-id: 02dd592719f41ad0320f065ccfe41dc64d030608
Summary:
The Thrift stats only expose the processing time of the Thrift handler, not of
the streaming part. In order to better understand the time actually spent in
streamChangeSince, let's collect data in it.
The code was written in such a way that we can easily expand it for all the
other Thrift requests, which can be done as a follow up change.
Reviewed By: chadaustin
Differential Revision: D37255869
fbshipit-source-id: 10b07b74d9516b58ab77bfc0a7eb3a650d13ec78
Summary:
On Windows, changing the case of a directory is a bit finicky. EdenFS currently
tries to remove the content of the directory, then calls PrjDeleteFile on it
and finally add the new directory. In theory, this has the effect of removing
the directory name from ProjectedFS cache, and then adding the differently
cased name.
This works well (as the existing tests shows), until an untracked file is
present in the directory. In order to not lose user data, EdenFS will not
remove it, but still try to call PrjDeleteFile on the directory. Unfortunately,
PrjDeleteFile will not be successful on a non-empty directory, thus the case
change fails. Worse, EdenFS believes that directory is empty, causing
discrepency in `hg status`.
Testing this with Mercurial shows that Mercurial will change the case of
directories only when there are no untracked files, and as soon as an untracked
file is present, the directory case will be unchanged.
Thus to solve this, we can first try to change the case, and if that fails,
fallback to just updating the directory content.
Reviewed By: chadaustin
Differential Revision: D36332663
fbshipit-source-id: bfa20fb703142a2b6090fe2718400781cc18ef2e
Summary:
Applies new import merging and sorting from µsort v1.0.
When merging imports, µsort will make a best-effort to move associated
comments to match merged elements, but there are known limitations due to
the diynamic nature of Python and developer tooling. These changes should
not produce any dangerous runtime changes, but may require touch-ups to
satisfy linters and other tooling.
Note that µsort uses case-insensitive, lexicographical sorting, which
results in a different ordering compared to isort. This provides a more
consistent sorting order, matching the case-insensitive order used when
sorting import statements by module name, and ensures that "frog", "FROG",
and "Frog" always sort next to each other.
For details on µsort's sorting and merging semantics, see the user guide:
https://usort.readthedocs.io/en/stable/guide.html#sorting
Reviewed By: lisroach
Differential Revision: D36402162
fbshipit-source-id: 6d180e9003d466c4f866fc9d454c6531766ca1dd
Summary:
As I'm trying to figure out how to trigger a case change bug, I wrote these 2
tests to exercise more of EdenFS case change code. They don't fail, but would
be good to keep around against future regressions.
Reviewed By: genevievehelsel
Differential Revision: D36086020
fbshipit-source-id: 35a7a89d67ae4a8623c97adb9788491b2f72aa39
Summary:
This implementation is the simplest possible which simply mimic what is being
done in Watchman. Changes are sent over the wire as they are detected and
thanks to Watchman ending the stream whenever it received enough, the diff code
will stop whenever the stream ends.
Reviewed By: chadaustin
Differential Revision: D35817807
fbshipit-source-id: 206a4b53383d9e5642539c4b2a70c3905405a9a6
Summary:
This makes it consistent with the naming in the CLI, and will allow introducing
a get_thrift_client for streaming APIs.
Reviewed By: genevievehelsel
Differential Revision: D35767106
fbshipit-source-id: deb45090875b0366111e16e27abba22858b8f370
Summary:
In Mercurial terminology, a `hg reset` operation simply updates the working
copy parent but doesn't affect the state of the repository. A diff/checkout
operation however would compare the on-disk state of the repository with the
working copy parent. Thus EdenFS must respect this behavior.
On Linux and macOS, a reset operation merely updates some state in the
EdenMount, but since these platforms reads data from the inode hierarchy, the
reset operation doesn't affect future reads to the repository. These would
still read the data from the previously checked out revision.
On Windows however, ProjectedFS requires us to read data from the currently
checked out commit, and thus we bypass the inode hierarchy. The downside is
that a reset operation changes the RootId in the EdenMount which is used to
walk Mercurial's tree hierarchy. What this means is that a read operation
performed after a reset would read the content of the file as it is in the
reset revision. This is not the behavior that Mercurial expects.
To solve this, we need to keep track of both the last checked out revision to
be used in response to read callbacks and the last reset revision, to be used
in diff operations. To achieve this, we can use the newly introduced SNAPSHOT
file format.
Reviewed By: fanzeyi
Differential Revision: D35293079
fbshipit-source-id: 5cfae510db8bc759d51447e6c24f021c104b7353
Summary: Do the same trick we for some other sparse commands to allow usage in eden repos.
Reviewed By: StanislavGlebik
Differential Revision: D35506423
fbshipit-source-id: b02f8f76f18acbbd24431da0d41b8322b9556658
Summary:
We have a few error reports of
```
Reviewed By: xavierd
Differential Revision: D35264311
fbshipit-source-id: 4a601a8b8da7aa89eede94060105ebffaf777e83
Summary:
We have a few error reports of
```
Reviewed By: xavierd
Differential Revision: D35262460
fbshipit-source-id: 4b9ceea8e275c97aeca25a1dad2f286f0311e547
Summary:
During a checkout operation, the server state executor is overwhelmed with
requests, with some of them on Windows waiting to take some overlay locks.
Unfortunately, reading the root tree involves several round trips to the same
executor which get queued after the checkout requests. In practice, what this
means is that a request to read the root tree while a checkout operation is
ongoing can be waiting for a very long time and since every ProjectedFS
operation starts by obtaining the root tree, these can block user commands for
a extended amount of time, giving the feeling that EdenFS is hung.
To solve this, we can simply hold a reference to the root tree in the
EdenMount, this root tree is only going being updated during a checkout or a
reset operation, and we can thus easily obtain the root tree then instead of
for every ProjectedFS operations.
Reviewed By: chadaustin
Differential Revision: D34831590
fbshipit-source-id: 7083b4bdfc72c23d8fe400411e31ad66d2d72624
Summary:
When a directory is renamed, EdenFS receives a single notification on that
directory. Unfortunately, due to the implementation of the fileNotification
function, this means the old directory would get removed, and the destination
directory would just get created. However, none of the files or other
directories would get added to the inode hierarchy.
To fix this, we need to make sure to recursively add the content of a directory
when it gets created as some files may be present in it.
Reviewed By: fanzeyi
Differential Revision: D34731270
fbshipit-source-id: 39b25441408c1e44ca7e62522699b053dd3e6c5d
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/106
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/107
Summary
* Add EdenFS builds on external CI now EdenSCM is good
* Mac builds on github actions by using brew for system dependencies
To make this work had to fix some path ordering issues with install directories for Linux and Mac, and generalise the homebrew path fixups we were doing for bison to all the used homebrew packages.
Previously Installed packages were being added after system paths, so our own installed thing might be ignored. On github these meant system python 3.9 was being used for hg tests rather than our specified 3.8 (this showed we have some test fails on python 3.9 with "SystemError: deallocated bytearray object has exported buffers", that are beyond the scope of this diff to fix)
Also needed to include the getdeps generated python into the generated edenscmdeps3.zip archive setup.py produces otherwise EdenFS tests failed to import thrift.Thrift
Eden tests are hanging when run externally about half way through, so disable them on github actions for now as this PR is already fairly large. They work when run locally on an internal devserver, so probably some bit of environment necessary is not defined in the test runner
Reviewed By: chadaustin
Differential Revision: D34116505
fbshipit-source-id: d0d628db5daabc28d0bd8997cd5c1bc885ed1e73
Summary:
When an update removes a directory but a materialized ignored file/directory
was present, EdenFS will rightfully not remove that ignored file/directory.
However, the parent directory would have been a placeholder on disk. And since
placeholder directories can never be made full[0], the directory keep its
placeholder. Listing this directory would thus issue an opendir callback, but
EdenFS would fail due to that directory not being present in the manifest.
Since this case can only happen in this case, let's simply return an empty
directory and let ProjectedFS merge it with the on-disk directory content.
[0]: https://docs.microsoft.com/en-us/windows/win32/projfs/cache-state
Reviewed By: genevievehelsel
Differential Revision: D33676621
fbshipit-source-id: bdbf769175a6471a1a709bdb0b03a45bd60ef366
Summary:
With Facebook having been renamed Meta Platforms, we need to change the license
headers.
Reviewed By: fanzeyi
Differential Revision: D33407812
fbshipit-source-id: b11bfbbf13a48873f0cea75f212cc7b07a68fb2e
Summary:
The EdenApi methods requiring `reponame`. It leaks the HTTP URL detail to many
more layers.
This diff makes the `Client` use the repo name provided by the config, or
explicitly via the `repo_name` API (as in tests). The HTTP client uses the repo
name in its config so callsites don't need to provide the name again.
Previously, if reponame is not provided, the HTTP client builder will use
"unknown" as fallback. That behavior is changed to an error instead.
The above change breaks some tests using production configs on scmstore
construction. scmstore is updated to properly avoid constructing EdenApi
if repo name or URL is not provided, instead of constructing an EdenApi
HTTP client using the wrong "unknown" URL silently.
This diff does not change the `EdenApi` trait yet, since that is a large change.
Reviewed By: yancouto
Differential Revision: D32800595
fbshipit-source-id: a156e18b2f6fc41804cc40e3c926086be63fd9b9
Summary:
The test started failing on CI due to the file not being read when being
locked. ProjectedFS behavior when the file is a placeholder vs a full file
differs when that file is locked and invalidated. In the case of a placeholder,
the file will be removed, while the full file won't. Let's force a full file by
simply reading it before locking it.
Reviewed By: kmancini
Differential Revision: D32776736
fbshipit-source-id: faf0bca1becc4bfed8cd6bc804e33a0540f3b8cb
Summary:
The new sparse profile config logic was triggering during an Eden code
path, where sparse doesn't exist. This caused a crash. Let's check for sparse
before trigging the path.
This only occured if there was another exception during checkout.
Reviewed By: quark-zju
Differential Revision: D32436881
fbshipit-source-id: 6d5da888e4f8a62a05a390c7e87bbc3061356fc2
Summary:
In order to compute the sha1 of materialized files in the working copy, EdenFS
on Windows will open and read the file out of the working copy directly. In the
case where EdenFS is wrong about the state of the file, this can lead to a
ProjectedFS callback to be triggered which may need to take the same lock that
the sha1 computation holds. This obviously results in a deadlock and to protect
against this, EdenFS would bail early when detecting recursive callbacks.
With read-only callbacks having been switched to only take into account the
underlying Mercurial data, and not the inode state, these callbacks will never
attempt to take any inode locks. This also means that the deadlock described
above can no longer happen: the ObjectStore layer will never attempt to read
anything from the working copy, thus recursive callbacks are impossible. These
checks are thus unecessary.
In theory, these checks could stay in place as a safety measure, but EdenFS
will soon need to list directory entry and call PrjGetOnDiskFileState in
response to file/directory creation/removal to handle the case where these can
happen out of order.
Reviewed By: genevievehelsel
Differential Revision: D32149411
fbshipit-source-id: 73b228bb8c73416b38f7f2042c25964e1aef3c4e
Summary:
In ProjectedFS terminology, the "backing store" refers to the state of files
and directories in the source control backend. For the longest time, EdenFS
assumed that the Inodes hierarchy was the "backing store", but this lead to
subtle bugs. For instance, renaming a non-hydrated placeholder would render the
file unreadable due to the read callback from ProjecedFS callback using the old
file name, not the new one
(https://github.com/microsoft/ProjFS-Managed-API/issues/68). It also meant that
files just created would always get a tombstone written on disk when being
removed as ProjectedFS queries EdenFS when removing a file via the access
callback. Returning true from this callback means that this file is present in
the backing store and thus ProjectedFS needs to create a tombstone on disk.
In order to fix these issues, EdenFS needs not to query the Inodes hierarchy,
but the Tree hierarchy. The latter matches with the expected behavior from
ProjectedFS.
For modification callbacks, these are by design acting on the Inode hierarchy
and thus are unchanged.
Reviewed By: chadaustin
Differential Revision: D32022639
fbshipit-source-id: bef18a0709a7eff9ef0a48a90126d1339be0023e
Summary:
This newly added config controls the default behavior of `eden clone` and
whether a new clone will use NFS or FUSE. This is intended to facilitate the
transition to NFS from FUSE on macOS.
Differential Revision: D30110056
fbshipit-source-id: ea6b493aa803297952b46434f6d11d8edf58e40b
Summary: 2 of the disabled are passing, and the last one just needed a simple tweak.
Reviewed By: fanzeyi
Differential Revision: D30029260
fbshipit-source-id: ad386632e8b6fbf9c76530c8ce2df7d38990bbbd
Summary:
The mode bits don't make a lot of sense on Windows, so let's use the plain
0o777 as that's what they will look like on Windows.
Reviewed By: chadaustin
Differential Revision: D30028892
fbshipit-source-id: ced737f1290438708536b732095cee18e1b1c073
