Summary: To be able to quickly find all activity related to given Sandcastle job as discussed in [this workplace post](https://fb.workplace.com/groups/2120196508269853/permalink/2899040243718805).
Reviewed By: kmancini
Differential Revision: D27541803
fbshipit-source-id: a55900064bbee92da902de785ebe0c0e8738c3a2
Summary:
Currently eden on startup prints:
```
Starting edenfs (dev build), pid 190
Opening local RocksDB store...
Opened RocksDB store in 0.073 seconds.
Could not parse config.json file: couldn't read /var/twsvcscm/local/.eden/config.json: No such file or directory
Skipping remount step.
Started edenfs (pid 190)
Logs available at /var/twsvcscm/local/.eden/logs/edenfs.log
```
Would be convenient if it also printed session id, to be able to query scuba straight away.
Reviewed By: chadaustin
Differential Revision: D27522665
fbshipit-source-id: d7d4cf6c97bc551061761f2653375f208e393498
Summary: Refactoring to make the following diff smaller.
Reviewed By: chadaustin
Differential Revision: D27522581
fbshipit-source-id: 8f858714fcbfe4b8f8b1c3678bb2003623abbd94
Summary:
Still trying to enable Scuba logging in ovrsource TD to figure out what caused Buck regression when we started working with multiple watchman instances.
Did not try previous fix of pgrp (deploying stuff on Sandcastle is not trivial), although now I'm not sure that pgrp was the issue. Hard to say.
But now I'm launching CI differently and observe different symptoms.
`eden start` command finishes in 30 seconds (according to logs), but Sandcastle waits for something for 10 minutes and then somebody kills `scribe_cat` and Sandcastle continues. I don't really know what that means, but there's another issue I discovered:
`scribe_cat` inherits a copy of stderr fs created during daemon startup.
This diff fixes the issue.
Reviewed By: kmancini
Differential Revision: D27494520
fbshipit-source-id: 069f4e9ea1efb553cf7a7f18e20ae92c27da808d
Summary:
I see in Sandcastle logs it was 10 minutes between `eden start` command and the following command.
So I'm curious, is it eden took 10 minutes to start or Sandcastle did something else but did not log it?
This little message in log will help to understand that.
Reviewed By: kmancini
Differential Revision: D27491709
fbshipit-source-id: 796c8db5abe49b056bd81b03ea57585a761c3cb6
Summary:
We call `setsid` too late, [on successful startup](https://fburl.com/diffusion/p88am07t), after `scribe_cat` binary launched.
As a result, `scribe_cat` does not belong to process group of eden, it belongs to the process group of launcher process, so it is killed by `timeout` command in ovrsource CI.
To deal with it, this diff proposes additionally calling `setpgrp` call early during initialization.
Reviewed By: chadaustin
Differential Revision: D27436474
fbshipit-source-id: acb168c1ab5b78b1598c34ebece88847a9a6480d
Summary:
In the case where EdenFS crashes, any client performing IO to EdenFS would be
stuck in the D state waiting for the server to be back up and running. Since on
restart EdenFS will chose different ports, this makes it hard/impossible to
actually recover and will keep some process unkillable, often time requiring
rebooting a host.
To avoid this, we can make the mount "soft" which aborts the IO after some time
has elapsed and the server isn't back up. This should enable us to kill these
processes and recover more gracefully.
Reviewed By: chadaustin
Differential Revision: D27402817
fbshipit-source-id: ff81a4360900c4c94665174e3c0cf63402f1533e
Summary:
During an `hg update`, all the loaded inodes are unloaded and forgotten, except
the ones that had a positive refcount, these are forgotten when FUSE calls the
`forget` API. For NFS, this `forget` API doesn't exist but the protocol relies
on file handle to stay valid for some time. The duration for which a file
handle need to stay alive depends on several factor. The root one needs to
always be valid, the rest relies on the lookup cache that the client keeps.
This lookup cache is invalidated when the mtime for the parent directory
changes, but mtime changes are often not immediately detected, as attributes
are cached client side with the `acdirmin` and `acregmin` for directories and
files.
For now, this diff doesn't attempt to deal with files being out of date right
after an `hg update`, it merely accounts for the NFS client being able to pass
old file handles to EdenFS. The astude reader will have noticed that
InodeNumber are never reclaimed for now, for that, a time-based mechanism will
need to be added to forget InodeNumbers that have expired.
Reviewed By: kmancini
Differential Revision: D27405295
fbshipit-source-id: af4a4ce9e31bfcc335608da91f0247b50ab87b3f
Summary: On macOS, the mount argument are encoded with XDR, let's add them first before using them.
Reviewed By: genevievehelsel
Differential Revision: D27306770
fbshipit-source-id: 727824f05d3874119858af60c263267adfb3e61e
Summary:
The returned value was never used by any callers, let's simply not return any
value.
Reviewed By: kmancini
Differential Revision: D27418015
fbshipit-source-id: 2a6f15eee01052cdfa9ae334c34e69f2f0a74407
Summary:
When EdenFS is mounted via FUSE, the changed directory and files are flushed so
the kernel is forced to re-read them to obtain new InodeNumber. When NFS is
used, EdenFS can't directly invalidate the client view of the FS, instead we
need to rely on when and how is this invalidated. On Linux, a file handle won't
be refreshed if its parent directory hasn't changed, which is done by looking
at the mtime of the directory by performing a GETATTR call. It is thus crucial
that the mtime is updated during an `hg update`.
Reviewed By: chadaustin
Differential Revision: D27403729
fbshipit-source-id: 1f7195c6c33e2a34c3bb73145f404d652302d828
Summary:
We can move the ifdef in the function itself. This makes the functions calling
these easier to read, and it's less likely for someone to forgot to add the
proper ifdef. If we also ever want to make these do something on Windows, it
will be easier to do too.
Reviewed By: chadaustin
Differential Revision: D27403730
fbshipit-source-id: c5b78ea7c7eb70eaf8d4974e5bec14296f91576f
Summary:
Non-list optional data can be present in some XDR description, let's special
case it so the intent is clear when declaring XDR datastructures.
Reviewed By: fanzeyi
Differential Revision: D27306768
fbshipit-source-id: 9d4d18bf8deff16f859c6d28a2579341dac8ee6f
Summary:
After receiving a network packet, it's possible that more than one fragment
were received as part of it. We thus need to service all of them before
returning.
This would typically be seen when running `rg` in the repository, which would
cause hangs due to some requests not being serviced as they would stay in the
iobuf queue until a new packet was received.
Reviewed By: kmancini
Differential Revision: D27194038
fbshipit-source-id: 3d81c797b5be7d0466d4acad7208f6a82593b4ca
Summary:
Computing the length of an iobuf chain can be expensive due to having to walk
its entirety. Thanksfully, IOBufQueue can cache the total length when data is
appended to it, which makes computing the length a constant operation.
Reviewed By: kmancini
Differential Revision: D27194037
fbshipit-source-id: af659c162ada61f2796bf407f419f5f15e918c02
Summary:
By moving the work to a background threadpool, we can more quickly go back to
servicing incoming NFS requests and thus allow more work to be done
concurrently. This would allow tools like ripgrep to being able to use multiple
cores to search in the code base.
Reviewed By: genevievehelsel
Differential Revision: D27194040
fbshipit-source-id: 7f1775ddaaa7eaf8776a06d05951cb936cd3fbb5
Summary:
This used to be very useful in the early stages, as a way to manually test the
code, now that the NFS procotol is pretty well defined and tests are actually
running, this has outlived its usefulness, let's simply remove the code.
Reviewed By: kmancini
Differential Revision: D27194039
fbshipit-source-id: af86edd9f438448209a7d14ba66c9b54d90a9594
Summary:
When I wrote the NFS code, I used `std::move` a bit too much, on datastructure
where moving them is equivalent to copying them. Instead, we can simply use
references, which makes the code shorter, and thus more efficient.
Reviewed By: kmancini
Differential Revision: D27172574
fbshipit-source-id: d9f06bf3f519e3539cf5cd0a0c4e4a49ef8009a8
Summary:
This should have been added in D27243075 (5a150e125a) but I forgot to run `hg add` and it
was thus not added...
Reviewed By: fanzeyi
Differential Revision: D27279169
fbshipit-source-id: 69807cc05fef33f51b2a491b66c2e8aeb7136deb
Summary:
While clang has no issue compiling this code, gcc appears to choke on it,
failing to compile. This is unfortunate as this means we need to hardcode the
size of the serialized datastructure and validate it with a test.
Reviewed By: fanzeyi
Differential Revision: D27243075
fbshipit-source-id: 5cd59921bbd5d5be4dfb22789942eb022dac5bbe
Summary: For the prefetch-profile feature, added a matching enable option to go with the existing disable option. Enable temporarily enables the prefetch-profile feature for cases where a user has already used disable.
Reviewed By: kmancini
Differential Revision: D27191711
fbshipit-source-id: cf9cb7d4054e9fcee66bdb5f205562ec79c2757e
Summary:
When copying files from Finder to EdenFS, Finder appears to be issuing a couple
of getxattr calls to EdenFS, however having these being denied by osxfuse
causes the copy to fail, resulting in a subpar experience.
Reviewed By: fanzeyi
Differential Revision: D27137328
fbshipit-source-id: 66bcc2e316d832385da05f0768da9f82ab36a374
Summary:
This is unused, and the Thrift file mentions that it should be fine to remove
in July 2020. It's now March 2021, time to kill it.
Reviewed By: chadaustin
Differential Revision: D26852134
fbshipit-source-id: 2872185edd834f889b78802210071d16b881e14c
Summary: It's best to test for a regular file by comparing dtype_t, not mode_t directly.
Reviewed By: chadaustin
Differential Revision: D26851735
fbshipit-source-id: 62d3833a53748ea9b45794ec6390d158e888c241
Summary:
EdenFS doesn't register itself against the portmap client, and on some system
where it is not started, it appears to not work reliably, crashing EdenFS early
at startup. For now, let's only build it when services need to be registered.
Reviewed By: genevievehelsel
Differential Revision: D27162906
fbshipit-source-id: cc2a8a588a756e54253da31f9bc00fbe4e5312d9
Summary:
In a bunch of places, the code assumes that an EdenMount is associated with a
Fuse channel. With NFS, that's no longer the case, thus let's make sure to
check the return value of mount->getFuseChannel(). In the case where it will
make sense to have something for NFS, I've either added an EDEN_BUG, or a TODO,
so we can come back to it later.
Reviewed By: chadaustin
Differential Revision: D26836431
fbshipit-source-id: c061b8f20199e5af3139a5003827f184f6eac8d4
Summary:
In the case where EdenFS would fail to initialize very early, the optional
server would be empty, and thus trying to dereference it would lead to EdenFS
crashing with no good error message. Let's simply test for this.
Reviewed By: genevievehelsel
Differential Revision: D27162907
fbshipit-source-id: a078b9995a94e5a86cf2893cbebfe7f5ca6d064e
Summary:
The NFS readdir turns out to be pretty similar to the FUSE one, with a couple
of differences. For one, it only populates the directory entry name, it also
puts a limit on the total size of the serialized result, including all the
NFS/XDR overhead.
It is not specified if the . and .. entries need to be returned, but since the
NFS spec is usually pretty explicit about these and makes it clear that this is
for the most part a client burden, I didn't add these. I may have to revisit
this later when I get to manually browse a repository.
Since the READDIR RPC doesn't populate any filehandle, the client will have to
issue a LOOKUP RPC for each entries, potentially leading to some
inefficiencies. A future diff will implement the READDIRPLUS to fix these.
Reviewed By: chadaustin
Differential Revision: D26802310
fbshipit-source-id: b821b57021d0c2dca33427975b1acd665173bc5c
Summary:
This simplifies a handful of tests and will make writing the READDIR RPC a bit
less magic when computing the amount of memory needed per entry.
Reviewed By: chadaustin
Differential Revision: D26802312
fbshipit-source-id: fc66cb68f721ed34c8f9879cdda2cd8db6ed8daa
Summary: This merely adds the types for the READDIR RPC.
Reviewed By: chadaustin
Differential Revision: D26802313
fbshipit-source-id: 634ff9b3f97dc4dba56d225c1fb9eae0a94c02d5
Summary:
Looking at the spec, READDIRPLUS appears to be more complex to implement than
READDIR, for now, let's force the use of READDIR. Future changes will have to
implement READDIRPLUS as that will likely be a perf improvement.
Reviewed By: chadaustin
Differential Revision: D26802311
fbshipit-source-id: cb784d74507e6c2c2ba4dc0aebe69cfcd69db40b
Summary:
This type is very specific to Fuse, let's make it obvious. The readdir method
has also been renamed as it is also very specific to Fuse.
Reviewed By: chadaustin
Differential Revision: D26802309
fbshipit-source-id: c2acdfd1c0006935c59b685fcda729e1bef88928
Summary:
When creating the .hg directory, Mercurial issues a SYMLINK RPC, thus let's
support it.
Reviewed By: kmancini
Differential Revision: D26785005
fbshipit-source-id: a760d55e6117cc3725444c604e3e4036f4a317b2
Summary:
This is mostly just copying /usr/include/linux/fuse.h from my devserver and
updating some flags in FuseChannel to display the new flags.
Reviewed By: chadaustin
Differential Revision: D27144667
fbshipit-source-id: 4854c6edd4c793ca707db26fecd11e2a3e9d7b75
Summary:
Some of our subprocess calls are running into dylib errors. The cause looks to
be related to our environment variables. We already have environment hygenics
for buck, so lets borrow this to use elsewhere.
This is to fix prefetch profile fetching on mac, but I ran into another error
when testing `eden du --clean`.
Reviewed By: genevievehelsel
Differential Revision: D27135268
fbshipit-source-id: 3955ddefc5e9ff60e966f63f7dc65ef737186464
Summary: Free the statement cache before trying to release the database. This is causing issues in `eden rm`.
Reviewed By: chadaustin
Differential Revision: D27104645
fbshipit-source-id: 74f4c4e73b048f58b83a28c2d280e5cb1c767048
Summary:
suppress the deprecated errors.
found something similar in here: https://stackoverflow.com/questions/1902021/suppressing-is-deprecated-when-using-respondstoselector
example failure:
https://www.internalfb.com/intern/buck/build/a3b550b8-4099-4f27-8975-5bfffd6447e5/
```
eden/fs/inodes/test/OverlayTest.cpp:730:1: error:
'InstantiateTestCase_P_IsDeprecated' is deprecated: INSTANTIATE_TEST_CASE_P is deprecated, please use INSTANTIATE_TEST_SUITE_P [-Werror,-Wdeprecated-declarations]
INSTANTIATE_TEST_CASE_P(
/Users/kuki/fbsource/third-party/googletest/googletest/include/gtest/gtest-param-test.h:507:38:
note: expanded from macro 'INSTANTIATE_TEST_CASE_P'
static_assert(::testing::internal::InstantiateTestCase_P_IsDeprecated(), \
/Users/kuki/fbsource/third-party/googletest/googletest/include/gtest/internal/gtest-internal.h:1209:1: note: 'InstantiateTestCase_P_IsDeprecated' has been explicitly marked deprecated here
GTEST_INTERNAL_DEPRECATED(
/Users/kuki/fbsource/third-party/googletest/googletest/include/gtest/internal/gtest-port.h:2215:59:
note: expanded from macro 'GTEST_INTERNAL_DEPRECATED'
#define GTEST_INTERNAL_DEPRECATED(message) __attribute__((deprecated(message)))
```
Reviewed By: mzlee
Differential Revision: D27037957
fbshipit-source-id: b12cc500441c9ed4ed72825475c57047fb0c2076
Summary: This diff replaces `SqliteStatement` in `TreeOverlayStore` with the new `PersistendSqliteStatement`.
Reviewed By: chadaustin
Differential Revision: D26656314
fbshipit-source-id: dbbfeddd52d3be2a1b5d3b6121d3694c41841fd5
Summary:
This diff makes `SqliteStatement` to construct persistent statement by default.
This change should be harmless if the caching assumption is not true. If the statement is not cached by the caller, it's probably on a non-critical path.
Reviewed By: chadaustin
Differential Revision: D26639365
fbshipit-source-id: 26af6ccdba036dc0fccf774e9ffb6bb2a014c5ff
Summary: This diff implements the newly added methods to `IOverlay` so we can build & test `SqliteOverlay` on devserver.
Reviewed By: chadaustin
Differential Revision: D25283647
fbshipit-source-id: 50d0c2dd47cd8f74c860a131ffad1d6342a86ea2
Summary:
Prefetch profiles has been tested to work on windows. Prefetching will be a little slower
on windows as process creation is more expensive, but it is still functional, and should
be ready for us to gather some early numbers on windows sandcastle.
Reviewed By: xavierd
Differential Revision: D27016292
fbshipit-source-id: ed0c8b732b1dfb8bf0a301ae5e76469d4e59f8d2
Summary:
We don't have data on how often this repo-corrupted event occurs, so
begin logging it.
Reviewed By: xavierd
Differential Revision: D26968638
fbshipit-source-id: 95e62a595f74f22fdec8a60ebb0671e53cbe199e
