Summary: While testing something for another change, I came across this overlooked typo.
Reviewed By: wez
Differential Revision: D22894060
fbshipit-source-id: 8aa48ef5da714650c974adcf8a34a542fdd4ed9e
Summary:
Avoid some overhead and complexity by storing BufVec as a
unique_ptr<IOBuf>. The complexity can be reintroduced if we ever find
FUSE splice support to be a performance win for us.
Reviewed By: kmancini
Differential Revision: D22710795
fbshipit-source-id: e58eedc0fb5cea9e9743ccd20d3e4e2b7cc5d198
Summary:
Previously we log a process to Scuba when it does 2000 (fetchThreshold_) fetchs, but then in Scuba all processes have fetch_count = 2000. In order to see how many fetches a process really did approximately, we log the same process to Scuba every time it does 2000 more fetches.
Note: this change could make the total count of fetch-heavy events in Scuba inaccurate, as we log the same process more than once. So when users want to see how many fetch-heavy events happened, instead of setting "type = fetch_heavy", they should set exactly "fetch_count = 2000".
Reviewed By: chadaustin
Differential Revision: D22867679
fbshipit-source-id: ae3c768a8d3b03628db6a77263e715303a814e3d
Summary: Since local store compaction is not a hard requirement for graceful restart, make this issue non blocking. We've seen some users fail restarts because they had compaction issues due to lack of space on their device. If we fail during the compaction stage, we should continue the restart anyway. This is also because there is a chance that the local store will clear columns that are no longer in use.
Reviewed By: chadaustin
Differential Revision: D22828433
fbshipit-source-id: 9a2aaec64e77c2d00089834fda8f8cffda472735
Summary:
TL:DR:
A codemod did something a bit unclean, so they added a lint. This will keep bugging us if we make changes here, so let's satisfy the linter.
More info:
`x.y_ref() = ...` and `*x.y_ref() = ...` are pretty much the same except `*x.y_ref() = ...` can throw for optional fields.
A codemod added a bunch of `*x.y_ref() = ...`, but after they didn't want people to copy paste this for optional fields so they added a lint that pops up on non optional fields too :(
https://fb.workplace.com/groups/thriftusers/permalink/509303206445763/
Reviewed By: chadaustin
Differential Revision: D22823686
fbshipit-source-id: b3b1b8a3b6b1f1245176be19c961476e4554a8e5
Summary:
Previously, fetch heavy event's cmdline was delimited by '\x00' when logged to Scuba. (for example: `grep--color=auto-rtest.`)
Now we replace \x00 with a space, so command name and args will be separated by space. ( `grep --color=auto -r test .` )
Reviewed By: kmancini
Differential Revision: D22772868
fbshipit-source-id: 4ab42e78c7bc786767eee3413b9586739a12e8ac
Summary:
This helps in understanding what's going on when some files disappear and/or
aren't flushed properly.
Reviewed By: fanzeyi
Differential Revision: D22833201
fbshipit-source-id: 09beb5796cb40c0a93107ee6a3a3497abb2578f0
Summary:
The tilde got dropped as part of the changes in D22672240 (be3683b1d4)
(an easy mistake to make!) and that renders this function less
useful.
Thankfully the caps display isn't a critical function; just for
some diagnostic printing.
Reviewed By: chadaustin
Differential Revision: D22847590
fbshipit-source-id: 716d7c7bd674260687fbc09e3dc94538359f98b3
Summary:
`log()` passes fsck bars to standard output, but it will also print the same message to the log with level DBG2. (example below)
```V0713 07:05:45.971511 3510654 StartupLogger.cpp:96] [====================>] 100%: fsck on /home/ailinzhang/eden-state/clients/dev-fbsource6/local
```
Since we don't want the log file to be messed up with fsck bars, we use `logVerbose()` with level DBG7.
Reviewed By: kmancini
Differential Revision: D22727965
fbshipit-source-id: 0700503af511030df2abbca4ad2fa1540995e919
Summary:
Unfortunately, using sqlite causes `edenfsctl prefetch` to take several orders
of magnitude more time than with rocksdb, tuning sqlite to be faster (at the
expense of reliability) still doesn't come closer to rocksdb. From profiling,
this is due to sqlite only allowing a single thread to read/write onto it
serializing all the work that EdenFS is doing.
A rework on the storage will be necessary to be able to get both good
performance, and reliability but that's a long term project, for now, make
Windows use rocksdb by default.
Reviewed By: wez
Differential Revision: D22819084
fbshipit-source-id: 62f397858ed547da30ef8a6346b767461dc53493
Summary:
As opposed to FUSE, ProjectedFS sends notifications for file/directory creation
after the fact, and for directory that means these will be visible on disk before
EdenFS may be aware of it. While EdenFS usually process it quickly, a heavily
multi-threaded application that tries to concurrently create a directory
hierarchy may end up sending notifications to EdenFS in a somewhat out of order
fashion.
Since this should be a very rare occurence, we make this a very slow path by
being optimistic and calling `getInode` first, and then only if that fails, we
aggressively create all the parent directories. During a buck build of ~1k
jobs, this happened only 3 times.
If we fully think this through, this change doesn't fully fix the race, as a
similar race can now happen when a create and remove/rename operations are
concurrent. However, a client performing these operations concurrently is
either aware that this is racy and should handle these properly, or is most
likely buggy. Both of these should significantly reduce the likelyhod of this
happening, thus, I'm leaving this unfixed for now.
To better understand how frequently this happens, I've added a stat counter.
For now, these aren't published to ODS, but this will be tackled later.
Reviewed By: wez
Differential Revision: D22783484
fbshipit-source-id: ea3aafc2f77b65d3967f697f68114921d5909137
Summary:
To avoid mistakes when refactoring, static_assert that the template
variant of sendReply and make_iovec are only used with POD types.
Reviewed By: genevievehelsel
Differential Revision: D22710742
fbshipit-source-id: 4557761f3946fe8969ce31a42502f64cc3298e1d
Summary:
Previously, we kept a list of weak pointers to RequestData instances
to track live FUSE calls, but the only thing we need is the FUSE
request header, so directly track that instead.
Reviewed By: kmancini
Differential Revision: D22710716
fbshipit-source-id: 3820acb314ac2db85b86de128fd082bc4871d9c6
Summary:
Now that we are explicit about whether the kernel caches must be
invalidated, we can remove a use of folly::RequestContext.
Reviewed By: kmancini
Differential Revision: D22710518
fbshipit-source-id: 4bd5267bf5dd3135adf33e4f4fa1ea2649816564
Summary:
Avoid the cost of dynamically querying whether we are in a FUSE
request handler or not by passing a flag.
Reviewed By: kmancini
Differential Revision: D22710480
fbshipit-source-id: 010bb8efee8074441aa20aab0eb12277452c5252
Summary:
Avoid the cost of dynamically querying whether we are in a FUSE
request handler or not by passing a flag.
Reviewed By: kmancini
Differential Revision: D22710452
fbshipit-source-id: 818035b72b793fa895147d9df3bb668d5b9c55f3
Summary:
Avoid the cost of dynamically querying whether we are in a FUSE
request handler or not by passing a flag.
Reviewed By: kmancini
Differential Revision: D22710422
fbshipit-source-id: 65b0737ad5f8ca74d12f2c657691d3751df4aa54
Summary:
Avoid the cost of dynamically querying whether we are in a FUSE
request handler or not by passing a flag.
Reviewed By: genevievehelsel
Differential Revision: D22710397
fbshipit-source-id: 7c62f45dfc227416c91070842a349b9d0c626cba
Summary:
We are unifying C++ APIs for accessing optional and unqualified fields:
https://fb.workplace.com/groups/1730279463893632/permalink/2541675446087359/.
This diff migrates code from accessing data members generated from unqualified
Thrift fields directly to the `field_ref` API, i.e. replacing
```
thrift_obj.field
```
with
```
*thrift_obj.field_ref()
```
The `_ref` suffixes will be removed in the future once data members are private
and names can be reclaimed.
The output of this codemod has been reviewed in D20039637.
The new API is documented in
https://our.intern.facebook.com/intern/wiki/Thrift/FieldAccess/.
drop-conflicts
Reviewed By: iahs
Differential Revision: D22764126
fbshipit-source-id: 67b1bc6d4a9135f594d78325cee8a194255bdcb8
Summary: This is unused, and won't ever be used.
Reviewed By: genevievehelsel
Differential Revision: D22744352
fbshipit-source-id: 9d20db608f972288eaf33e3ea0a79ffe5e13e03e
Summary: These were only used in tests, no need to keep it.
Reviewed By: chadaustin
Differential Revision: D22744353
fbshipit-source-id: 57596d641ab85f15e8c945327d7849a64aa73ef8
Summary:
Both unices and Windows needs to invalidate the cache in the same place, let's
avoid ifdef and consolidate the function name to clean things up a bit.
Reviewed By: chadaustin
Differential Revision: D22741709
fbshipit-source-id: 04060c0080eff9840abd22747ea48404fa50fd86
Summary:
We're experimenting with enabling NO_OPEN support in our internal
build of the osxfuse kext. This commit includes the relevant capability bits
for the kernel interface (which are compatible with the linux FUSE
implementation) as well as adjusts our FUSE client code to detect and use
those bits on macOS.
Reviewed By: xavierd
Differential Revision: D22744378
fbshipit-source-id: 21376439a85b0b0f5a71916dd1af618d9627695e
Summary:
Even though we never asked for a pre rename notification, ProjectedFS would
anyway send one to us, and since we're now failing on unrecognized
notification, that meant renames would always fail. Handle them and do nothing in them.
I'm not sure yet if we want to move the actual rename logic into the pre rename
callback, or keep it as is. The benefit of the pre rename code is that it can
fail the user request, while the current code will fail, but not prevent the
rename from happening.
Reviewed By: wez
Differential Revision: D22738388
fbshipit-source-id: 487e1f90b503bc59cff7315dd38d2a3039552eaf
Summary: This will make it easier to support PrjFS async programming model.
Reviewed By: chadaustin
Differential Revision: D22738408
fbshipit-source-id: 2e2e4f6f2f718b0226cf9fab66589c50b6db49db
Summary:
The pattern when using it is to catch the exception, then call the function,
which then would re-throw it and catch the subtyped exception. We can avoid a
re-throw entirely by having the function take the exception that was thrown in
the first place and dynamic cast it.
Reviewed By: chadaustin
Differential Revision: D22736772
fbshipit-source-id: 0efa3134bccf3ba8bdcd51d67e03c7ee4483a99f
Summary:
We're still reading the entire file at a time, but this paves the way to not do
that.
While the change looks big, a lot of it is just moving code around. The main
gist of it is removing EdenMount::readFile and writing the proper future
combinator in EdenDispatcher::getFileData.
Reviewed By: wez
Differential Revision: D22361748
fbshipit-source-id: 6391a29d25a4c9e61b91952c40c21ad52e728c8b
Summary:
This would enable the enumeration callback to use ProjectedFS asynchronous
completion.
Reviewed By: chadaustin
Differential Revision: D22361747
fbshipit-source-id: b2d31533ee5128e9dd3da7f91d5225331cf8e926
Summary:
We've had a long latent bug where for some reason a file that should be in the
repo isn't visible on the filesystem. Turning on logging and running `ls` shows
that EdenFS tells ProjectedFS about the file, but it's somehow not shown. The
reason being that ProjectedFS keeps track of all the removed files by adding a
tombstone for these, and will not list them until EdenFS removes this
tombstone.
Looking at the various places where we invalidate FUSE's cache, but not
ProjectedFS, only one stands out, and it has to do with files not being present
in the working copy, but added in the update destination. Adding the right
flush there appear to solve the simple repro listed below.
I'll remove all the ifdef around flushing in a later diff to avoid this issue
from re-appearing again.
Reviewed By: fanzeyi
Differential Revision: D22739916
fbshipit-source-id: 3a4fbc825cd21b36cbd2616882fd50e3d9741f63
Summary:
This adds logging for data fetches that come from the thrift globfiles call to
help debug the cause of unexpected data fetches. (See D22448048 for more
motivation)
Reviewed By: genevievehelsel
Differential Revision: D22489512
fbshipit-source-id: 040cf1277205d08ea864a1f30f8d3b25ee7c4508
Summary:
This adds logging for files fetched in prefetch like was aleady added for
blob and tree fetches.
This is needed to log the fetches caused by the glob files thrift call. The
purpose of this to help debug the cause of unexpected data fetches (See
D22448048 for more motivation).
Reviewed By: genevievehelsel
Differential Revision: D22561619
fbshipit-source-id: 5ae78b99fb0c7d863d8223b93492b0d0210ddf9e
Summary:
This adds logging for data fetches that come from the thrift getSHA1 call to
help debug the cause of unexpected data fetches. (See D22448048 for more
motivation)
Reviewed By: genevievehelsel
Differential Revision: D22489514
fbshipit-source-id: eb2d82c206af857cc79439d2854d682641292db8
Summary:
This adds logging for data fetches that come from the thrift getFileInformation call to
help debug the cause of unexpected data fetches. (See D22448048 for more
motivation)
Reviewed By: genevievehelsel
Differential Revision: D22489513
fbshipit-source-id: ab6283476d05b06b9f9e37c6b4fd81c1282046ff
Summary:
This adds logging for data fetches that come from the thrift checkout call to
help debug the cause of unexpected data fetches. (See D22448048 for more
motivation)
Reviewed By: chadaustin
Differential Revision: D22489504
fbshipit-source-id: 3b732a1e5627c2130f561ec0138a1df270e1925d
Summary:
We have seen that some of the unexpected data fetches do not originate from
FUSE. This adds parity to the logging for data fetches that come from the thrift
interface. Adding this logging improves the overall observability of eden, and
will help us debug the cause of unexpected data fetching.
This introduces plumbing to allow logging data fetches that originate from
thrift requests.
Reviewed By: chadaustin
Differential Revision: D22448048
fbshipit-source-id: a39dde72467c4922c07c569c14fb499341d40258
Summary:
What does this do: allocate a ThriftLogHelper and ensure it lives for the
duration of its coresponding thrift request.
Why: By ensuring that the log helper's lifetime matches that of the thrift
request, we simplify adding to the logger. We wont have to think about the
lifetime of each of the members of the logger, we simply can stick them in the
log helper and let shared_ptr do its magic.
More specific why: This refactor helps make adding Thrift data fetch logging
cleaner (see following changes). We need to ensure the ObjectFetchContext
stays around for the duration of the thrift request since we pass it around
by reference. Sticking this in ThriftLogHelper avoids adding another disjoint
piece of plumbing and makes the code easier to maintain going forward.
Reviewed By: genevievehelsel
Differential Revision: D22632546
fbshipit-source-id: 1baa79419386947e52a386d89a65f032f1988622
Summary: This is unused, no need to keep it around.
Reviewed By: wez
Differential Revision: D22361749
fbshipit-source-id: 3c353776437b59c6c7735652f7eb1ce052215e11
Summary:
Instead of overloading the FileMetadata struct that is used for directory
listing, let's use one tailored for our use. This will enable more flexibility
when deciding what to provide ProjectedFS. For instance, we could send the
InodeNumber so we don't need to do expensive path resolution for every notification.
Reviewed By: wez
Differential Revision: D22361751
fbshipit-source-id: 4801be45d8afc3af51e0a9564d9acb0a8e32255a
Summary:
Futures are intended to be chained together and not synchronously waited one
after the other. While this may be one of the goal, it also paves the way to
enable ProjectedFS asynchronous notification handling.
While doing this, a bunch of code was moved from EdenMount.cpp to the
dispatcher itself, the rationale behind this is to follow what the unix
EdenDispacher with the long term plan to merge the 2 as much as possible.
Reviewed By: wez
Differential Revision: D22361750
fbshipit-source-id: fa679a8b94ff6f8b5a33782fdb6b129ab066c4d8
Summary:
Whenever EdenFS reads a file from the overlay, there is a risk of this
triggering a prjfs notification, especially when there is a discrepancy between
the actual state of the overlay, and what EdenFS thinks the overlay should look
like.
A simple example of this includes removing a materialized file in the overlay
when EdenFS isn't running, and running `hg status` when it's back up. The
status call will try to compute the sha1 of the file, and thus will call into
the `CreateFile` API, which will then trigger an EdenFS callback to be called.
This callback will then try to recursively acquire the inode lock, which is
already held by the sha1 computation code. And that's a textbook example of a
deadlock.
To avoid this in the first place, let's simply refuse recursive callbacks. The
sha1 code will either have to be modified to handle errors, or we need to scan
the overlay when mounting the repo.
Reviewed By: fanzeyi
Differential Revision: D22288431
fbshipit-source-id: 2b4b31eddf7debcde55c22dd704d198ec44e59b4
Summary:
At mount time, EdenFS will try to create the .eden/config file which will
indicate to the client that this is an EdenFS repo, this config also contains
the location of the socket to talk to EdenFS.
On unices, and while the .eden directory is slightly different, the content of
this directory is written onto the overlay, but on Windows, the overlay is the
repository itself. What this means is that creating this config file will end
up triggering a ProjFS callback, which can potentially lead to deadlocks if
we're not careful.
A future change will simply prevent these recursive callbacks from happening
and thus a solution needs to be found for the .eden/config file itself. Since
the file itself is tiny (about 400B), and should only be accessed once[0], the
simple approach to keep it in memory and special case it works perfectly.
[0]: Once a file was read fully by ProjFS, it's present in the overlay and
EdenFS will no longer be requested to provide it.
Reviewed By: chadaustin
Differential Revision: D22310734
fbshipit-source-id: 6b2dba2164496ebd251104d7875b51569be2471f
Summary: Previously, `BackingStore` and all its sub-classes' `getBlob` and `getTree` methods accepted both `ObjectFetchContext` and `ImportPriority` as arguments. Now, `ImportPriority` is removed because we can get the priority from `ObjectFetchContext `
Reviewed By: kmancini
Differential Revision: D22650629
fbshipit-source-id: e1b0c57a059f11504b28b2c17d698bb58f51e1ee
Summary:
check fetch count before `getPriority()` is used. If fetch count has exceeded `fetchThreshold_`, lower the priority by 1.
Note: this diff only guarantees that the `getPriority()` function is returning the lowered priority. How the returned value is used for scheduling is handled by `HgQueuedBackingStore`
Reviewed By: kmancini
Differential Revision: D22550640
fbshipit-source-id: c032f8f72ca658618ac118dfb3ad3dcae61e9735
Summary: Previously `getPriority()` always returned a fixed priority. Now that we want `ObjectFetchContext` to lower its priority, it is better to make it have its own copy of `ImportPriority`.
Reviewed By: kmancini
Differential Revision: D22550587
fbshipit-source-id: 029c797def477ae4533f66cfba146a3972cfb65d
Summary: This diff ensures that the value of ImportPriority is always positive and changes offset from 0 to half of the maximum offset to allow lowering offset in the future.
Reviewed By: kmancini
Differential Revision: D22550462
fbshipit-source-id: 69f45369359c7b2c39a0c6831e9b33982e26a16a
Summary:
within() schedules work on the singleton timekeeper thread which
sometimes scheduled a timer callback on the ServerExecutor while
TestMount was shutting down, causing an assertion failure. Since
TestMount's executor is deterministic, just make the entire test
deterministic.
Reviewed By: genevievehelsel
Differential Revision: D22709150
fbshipit-source-id: 0a1621df0a9934bb7944fc259f7f83d110b35e93
Summary: If the user specifies a mount, only clear for that mount, otherwise clear for all mounts.
Reviewed By: fanzeyi
Differential Revision: D22674210
fbshipit-source-id: 832c4fd37a63fdb44cb12844378b61d113731016
Summary: add a thrift call to clear `pidFetchCount_` in `ObjectStore` and call it in `eden debug gc_process_fetch`. Users might want this command to start a new recording of process fetch counts.
Reviewed By: kmancini
Differential Revision: D22583430
fbshipit-source-id: eba7d63b08da5134fd09b7512895aba06f6a7ca5
Summary:
In Windows 10 2004, the behavior of AF_UNIX socket changed slightly, a socket
on which a server is listening to now appears to be shown as a "reparse point"
on disk. This appears to confuse Python as `os.path.exists(socketpath)` no
longer returns `True` for a listened to socket, which means that edenfsctl
would not be able to communicate with EdenFS, even though it is up and running.
Windows still has this weird behavior where the socket is created on disk
during connect, but EdenFS knows about this and already retries several times
(D21755498 (2c5768404d)) to avoid the issue, it's therefore fine to remove this code.
Reviewed By: wez
Differential Revision: D22666081
fbshipit-source-id: e2eea3fdd7628354b8b187b76899be4e9e2fa17f
