Summary:
activate recently got broken when we added the prefetch-metadata flag,
this needs to be on activate as well as fetch
Reviewed By: xavierd
Differential Revision: D27778771
fbshipit-source-id: 052710c2f206e66d8042314773b6b408cff4915c
Summary: This seems to have broken the EdenFS HgPrefetch test.
Reviewed By: xavierd
Differential Revision: D27795192
fbshipit-source-id: 80a748036961aa6a5750182bf65637fb76825341
Summary:
Previously we'd skip dynamicconfigs when there wasn't a repo available.
Now that dynamicconfig can represent the NoRepo state, let's load dynamicconfigs
in that situation.
This also supports the case where there is no user name.
Reviewed By: kulshrax
Differential Revision: D26801059
fbshipit-source-id: 377cfffb6695a7fbe31303868a88862259ebf8c4
Summary: Since HeaderClientChannel now accepts a transport unique_ptr there's no need to have this deleter exposed outside of HeaderClientChannel.
Reviewed By: iahs
Differential Revision: D27729209
fbshipit-source-id: 064b03afdfe567b6df6437348596f0f6f97f6aaf
Summary:
This allows unix sockets to be created in the mount. This will allow Buck to
run properly as it tries to create sockets in the repository.
Reviewed By: kmancini
Differential Revision: D27690406
fbshipit-source-id: 5725d68bdda12f3a5882ce48b6bdd02b14cdece4
Summary: This merely adds the types for the procedure
Reviewed By: kmancini
Differential Revision: D27690405
fbshipit-source-id: b94fb03658cabaece4166c29135c5fdf9a613d3c
Summary:
This is roughly the same logic as the UNLINK one with the only difference being
in the handling of "." and "..".
Reviewed By: kmancini
Differential Revision: D27684716
fbshipit-source-id: 86a95c38e6c783bc3a45c0a8b000d0210b6dd0b8
Summary:
This merely adds the types needed for the RMDIR procedure. Implementation will
follow.
Reviewed By: genevievehelsel
Differential Revision: D27684736
fbshipit-source-id: 84f5a4f3dc805e7893853b0de1dc19cb01c1319f
Summary:
To get the size in bytes, we need to multiply the quantity with the block size,
not with itself.
Reviewed By: genevievehelsel
Differential Revision: D27690857
fbshipit-source-id: 7d7ca767881b1118fc24befed230a63f342bc911
Summary:
MacOS does not have the device field like linux that we can use to mark edenfs
nfs mounts. But there is the `f_mntfromname` field. This field more typically
might have the path which this nfs mount is mirrored from, but it should be fine
to hyjack as the edenfs indicator field.
Reviewed By: xavierd
Differential Revision: D27717945
fbshipit-source-id: 056fb39dc3273b68d79c26487fd045f4e7f93b7b
Summary:
With fuse we report "edenfs:" as the device, let's do the same thing with nfs
so watchman can recognize edenfs nfs mounts similarly.
I think its fine to use the standard "edenfs" as the server name in the mount
call rather than the address, from looking at:
https://www.systutorials.com/docs/linux/man/8-mount.nfs/
Reviewed By: xavierd
Differential Revision: D27630764
fbshipit-source-id: 9e476c90ece90e758b98d140c6bf4067dbca3661
Summary: Running these with tsan appears to run properly, let's try to re-enable them.
Reviewed By: genevievehelsel
Differential Revision: D27723525
fbshipit-source-id: 42e61d26cf478cbe808698a6a0615015832180fa
Summary:
The CMake documentation states:
>By default, the value index of value-parameterized tests is replaced by the
>actual value in the CTest test name. If this behavior is undesirable (e.g.
>because the value strings are unwieldy), this option will suppress this
>behavior.
Which appears to be a decent default, but not when the parameter is a pointer,
in which case the test name will contain some hex values.
Reviewed By: genevievehelsel
Differential Revision: D27713222
fbshipit-source-id: 0f15b24d04817384ff975ad7b07e16b744e1eb2e
Summary:
It looks like nfs_test is tripping the nightly pyre infer job causing it to not
publish diff that increase our typing coverage. Let's manually type it to
unblock the nightly job.
Reviewed By: genevievehelsel
Differential Revision: D27682093
fbshipit-source-id: a32df9f5b8eeaef2006de7d64f5adadb763402e8
Summary: Migrate the codebase away from the ad-hoc `folly::uint64ToBufferUnsafe` and to `folly::to_ascii_decimal` which is intended for these cases.
Reviewed By: WillerZ
Differential Revision: D27281875
fbshipit-source-id: 0c98749e4aed9c873853eed2221cf54a89279ff4
Summary:
We are moving our skelton for eden top a bit further, by creating the view
structure for eden top. This creates the widget for the banner of eden top,
the help page and the main page as well as a couple sections of the main
page.
The main page is displayed on default and they are toggled with `h` for help
page and `esc` to return to the mage page. Visable and hidden widgets are not
implemented in termwiz yet so we have to do a bit of hacking to hide and
display widgets for our selves.
Each of the sections is stubed with place holder text for testing.
Reviewed By: xavierd
Differential Revision: D26892620
fbshipit-source-id: a7bb4d0e11f3a8968ef071e7f585d07a9c286880
Summary:
Our logs are filled with this message, making it almost impossible to read,
this doesn't need to be DBG1, let's reduce it to DBG5 similarly to what we do
for blobs.
Reviewed By: kmancini
Differential Revision: D27652526
fbshipit-source-id: c61d502a9f17d8180bad857ff6c1a9033763947d
Summary: Here we further add prefetch-metadata support to prefetching profiles
Reviewed By: genevievehelsel
Differential Revision: D27568542
fbshipit-source-id: 64507125f47cf093c0133c82fcab941ed6495f32
Summary:
On macOS, the mount syscall for NFS expects the arguments to be XDR encoded.
This set of argument roughly match with its Linux counterpart and appears to
start the mount process. It fails early on when trying to access the .hg
directory but this is probably an issue with the NFS server code, not of the
mounting code.
Reviewed By: kmancini
Differential Revision: D27306769
fbshipit-source-id: 697fadfddc4048ef56c3a23f75dd5bdbcc92af1b
Summary:
* use `std::nullopt`
* TODO about sandcastle_instance_id in opensource version
Reviewed By: chadaustin
Differential Revision: D27575732
fbshipit-source-id: bf76970a15fee5a3dc1e4e411ea70f5af7248496
Summary: To be able to quickly find all activity related to given Sandcastle job as discussed in [this workplace post](https://fb.workplace.com/groups/2120196508269853/permalink/2899040243718805).
Reviewed By: kmancini
Differential Revision: D27541803
fbshipit-source-id: a55900064bbee92da902de785ebe0c0e8738c3a2
Summary:
Currently eden on startup prints:
```
Starting edenfs (dev build), pid 190
Opening local RocksDB store...
Opened RocksDB store in 0.073 seconds.
Could not parse config.json file: couldn't read /var/twsvcscm/local/.eden/config.json: No such file or directory
Skipping remount step.
Started edenfs (pid 190)
Logs available at /var/twsvcscm/local/.eden/logs/edenfs.log
```
Would be convenient if it also printed session id, to be able to query scuba straight away.
Reviewed By: chadaustin
Differential Revision: D27522665
fbshipit-source-id: d7d4cf6c97bc551061761f2653375f208e393498
Summary: Refactoring to make the following diff smaller.
Reviewed By: chadaustin
Differential Revision: D27522581
fbshipit-source-id: 8f858714fcbfe4b8f8b1c3678bb2003623abbd94
Summary:
Still trying to enable Scuba logging in ovrsource TD to figure out what caused Buck regression when we started working with multiple watchman instances.
Did not try previous fix of pgrp (deploying stuff on Sandcastle is not trivial), although now I'm not sure that pgrp was the issue. Hard to say.
But now I'm launching CI differently and observe different symptoms.
`eden start` command finishes in 30 seconds (according to logs), but Sandcastle waits for something for 10 minutes and then somebody kills `scribe_cat` and Sandcastle continues. I don't really know what that means, but there's another issue I discovered:
`scribe_cat` inherits a copy of stderr fs created during daemon startup.
This diff fixes the issue.
Reviewed By: kmancini
Differential Revision: D27494520
fbshipit-source-id: 069f4e9ea1efb553cf7a7f18e20ae92c27da808d
Summary:
I see in Sandcastle logs it was 10 minutes between `eden start` command and the following command.
So I'm curious, is it eden took 10 minutes to start or Sandcastle did something else but did not log it?
This little message in log will help to understand that.
Reviewed By: kmancini
Differential Revision: D27491709
fbshipit-source-id: 796c8db5abe49b056bd81b03ea57585a761c3cb6
Summary:
We call `setsid` too late, [on successful startup](https://fburl.com/diffusion/p88am07t), after `scribe_cat` binary launched.
As a result, `scribe_cat` does not belong to process group of eden, it belongs to the process group of launcher process, so it is killed by `timeout` command in ovrsource CI.
To deal with it, this diff proposes additionally calling `setpgrp` call early during initialization.
Reviewed By: chadaustin
Differential Revision: D27436474
fbshipit-source-id: acb168c1ab5b78b1598c34ebece88847a9a6480d
Summary:
In the case where EdenFS crashes, any client performing IO to EdenFS would be
stuck in the D state waiting for the server to be back up and running. Since on
restart EdenFS will chose different ports, this makes it hard/impossible to
actually recover and will keep some process unkillable, often time requiring
rebooting a host.
To avoid this, we can make the mount "soft" which aborts the IO after some time
has elapsed and the server isn't back up. This should enable us to kill these
processes and recover more gracefully.
Reviewed By: chadaustin
Differential Revision: D27402817
fbshipit-source-id: ff81a4360900c4c94665174e3c0cf63402f1533e
Summary:
During an `hg update`, all the loaded inodes are unloaded and forgotten, except
the ones that had a positive refcount, these are forgotten when FUSE calls the
`forget` API. For NFS, this `forget` API doesn't exist but the protocol relies
on file handle to stay valid for some time. The duration for which a file
handle need to stay alive depends on several factor. The root one needs to
always be valid, the rest relies on the lookup cache that the client keeps.
This lookup cache is invalidated when the mtime for the parent directory
changes, but mtime changes are often not immediately detected, as attributes
are cached client side with the `acdirmin` and `acregmin` for directories and
files.
For now, this diff doesn't attempt to deal with files being out of date right
after an `hg update`, it merely accounts for the NFS client being able to pass
old file handles to EdenFS. The astude reader will have noticed that
InodeNumber are never reclaimed for now, for that, a time-based mechanism will
need to be added to forget InodeNumbers that have expired.
Reviewed By: kmancini
Differential Revision: D27405295
fbshipit-source-id: af4a4ce9e31bfcc335608da91f0247b50ab87b3f
Summary: On macOS, the mount argument are encoded with XDR, let's add them first before using them.
Reviewed By: genevievehelsel
Differential Revision: D27306770
fbshipit-source-id: 727824f05d3874119858af60c263267adfb3e61e
Summary:
The returned value was never used by any callers, let's simply not return any
value.
Reviewed By: kmancini
Differential Revision: D27418015
fbshipit-source-id: 2a6f15eee01052cdfa9ae334c34e69f2f0a74407
Summary:
When EdenFS is mounted via FUSE, the changed directory and files are flushed so
the kernel is forced to re-read them to obtain new InodeNumber. When NFS is
used, EdenFS can't directly invalidate the client view of the FS, instead we
need to rely on when and how is this invalidated. On Linux, a file handle won't
be refreshed if its parent directory hasn't changed, which is done by looking
at the mtime of the directory by performing a GETATTR call. It is thus crucial
that the mtime is updated during an `hg update`.
Reviewed By: chadaustin
Differential Revision: D27403729
fbshipit-source-id: 1f7195c6c33e2a34c3bb73145f404d652302d828
Summary:
We can move the ifdef in the function itself. This makes the functions calling
these easier to read, and it's less likely for someone to forgot to add the
proper ifdef. If we also ever want to make these do something on Windows, it
will be easier to do too.
Reviewed By: chadaustin
Differential Revision: D27403730
fbshipit-source-id: c5b78ea7c7eb70eaf8d4974e5bec14296f91576f
Summary:
Non-list optional data can be present in some XDR description, let's special
case it so the intent is clear when declaring XDR datastructures.
Reviewed By: fanzeyi
Differential Revision: D27306768
fbshipit-source-id: 9d4d18bf8deff16f859c6d28a2579341dac8ee6f
Summary:
After receiving a network packet, it's possible that more than one fragment
were received as part of it. We thus need to service all of them before
returning.
This would typically be seen when running `rg` in the repository, which would
cause hangs due to some requests not being serviced as they would stay in the
iobuf queue until a new packet was received.
Reviewed By: kmancini
Differential Revision: D27194038
fbshipit-source-id: 3d81c797b5be7d0466d4acad7208f6a82593b4ca
Summary:
Computing the length of an iobuf chain can be expensive due to having to walk
its entirety. Thanksfully, IOBufQueue can cache the total length when data is
appended to it, which makes computing the length a constant operation.
Reviewed By: kmancini
Differential Revision: D27194037
fbshipit-source-id: af659c162ada61f2796bf407f419f5f15e918c02
Summary:
By moving the work to a background threadpool, we can more quickly go back to
servicing incoming NFS requests and thus allow more work to be done
concurrently. This would allow tools like ripgrep to being able to use multiple
cores to search in the code base.
Reviewed By: genevievehelsel
Differential Revision: D27194040
fbshipit-source-id: 7f1775ddaaa7eaf8776a06d05951cb936cd3fbb5
Summary:
This used to be very useful in the early stages, as a way to manually test the
code, now that the NFS procotol is pretty well defined and tests are actually
running, this has outlived its usefulness, let's simply remove the code.
Reviewed By: kmancini
Differential Revision: D27194039
fbshipit-source-id: af86edd9f438448209a7d14ba66c9b54d90a9594
Summary:
When I wrote the NFS code, I used `std::move` a bit too much, on datastructure
where moving them is equivalent to copying them. Instead, we can simply use
references, which makes the code shorter, and thus more efficient.
Reviewed By: kmancini
Differential Revision: D27172574
fbshipit-source-id: d9f06bf3f519e3539cf5cd0a0c4e4a49ef8009a8
Summary:
This should have been added in D27243075 (5a150e125a) but I forgot to run `hg add` and it
was thus not added...
Reviewed By: fanzeyi
Differential Revision: D27279169
fbshipit-source-id: 69807cc05fef33f51b2a491b66c2e8aeb7136deb
Summary:
While clang has no issue compiling this code, gcc appears to choke on it,
failing to compile. This is unfortunate as this means we need to hardcode the
size of the serialized datastructure and validate it with a test.
Reviewed By: fanzeyi
Differential Revision: D27243075
fbshipit-source-id: 5cd59921bbd5d5be4dfb22789942eb022dac5bbe
Summary: For the prefetch-profile feature, added a matching enable option to go with the existing disable option. Enable temporarily enables the prefetch-profile feature for cases where a user has already used disable.
Reviewed By: kmancini
Differential Revision: D27191711
fbshipit-source-id: cf9cb7d4054e9fcee66bdb5f205562ec79c2757e
Summary:
When copying files from Finder to EdenFS, Finder appears to be issuing a couple
of getxattr calls to EdenFS, however having these being denied by osxfuse
causes the copy to fail, resulting in a subpar experience.
Reviewed By: fanzeyi
Differential Revision: D27137328
fbshipit-source-id: 66bcc2e316d832385da05f0768da9f82ab36a374
Summary:
This is unused, and the Thrift file mentions that it should be fine to remove
in July 2020. It's now March 2021, time to kill it.
Reviewed By: chadaustin
Differential Revision: D26852134
fbshipit-source-id: 2872185edd834f889b78802210071d16b881e14c
Summary: It's best to test for a regular file by comparing dtype_t, not mode_t directly.
Reviewed By: chadaustin
Differential Revision: D26851735
fbshipit-source-id: 62d3833a53748ea9b45794ec6390d158e888c241
Summary:
EdenFS doesn't register itself against the portmap client, and on some system
where it is not started, it appears to not work reliably, crashing EdenFS early
at startup. For now, let's only build it when services need to be registered.
Reviewed By: genevievehelsel
Differential Revision: D27162906
fbshipit-source-id: cc2a8a588a756e54253da31f9bc00fbe4e5312d9
