Summary:
after `startRecordingFetch()` is called by `HgQueuedBackingStore`, record the path for each fetched file. When `stopRecordingFetch()` is called, stop recording and return the collected file paths.
`startRecordingFetch()` and `stopRecordingFetch()` will be used in the next diff.
Reviewed By: chadaustin
Differential Revision: D22797037
fbshipit-source-id: a1fe30424d3c2884ffe139a0062b1e36328fd4fe
Summary:
Our error handling looked pretty, but allocating all of these futures
is expensive. Each future is an allocation and some atomics. This diff
buys back some performance which I will soon spend on a new async
event queue.
Reviewed By: xavierd
Differential Revision: D22799737
fbshipit-source-id: 91dcfe974cf8f461109dfaa9dbf75c054ed84f59
Summary:
I've seen the error a couple of times when messing up with my clones, not
having the path makes it a bit difficult to fully understand what's going on,
make sure we log it.
Reviewed By: fanzeyi
Differential Revision: D22899098
fbshipit-source-id: c9a60b71ea20514158e62fe8fa9c409d6f0f37ff
Summary:
Cache invalidation is hard, and on Windows we avoided doing a lot of them. It
turns out, this was the wrong decision as it's fairly easy to find cases where
the filesystem view is different from the manifest state.
Since the Linux code is most likely correct in where the invalidation is done,
let's also do the same on Windows, removing a whole lot of #ifdef. It is very
likely that as a result of this diff we end up invalidating more than needed,
thus slowing down EdenFS, but at this point I'd prefer to err on the side of
correctness, performance will come later.
While invalidating files should use PrjDeleteFile, for directories, we simply
need to mark them as placeholder, as directories created by a user won't have a
placeholder, thus ProjectedFS would bypass EdenFS when listing in.
Reviewed By: chadaustin
Differential Revision: D22833202
fbshipit-source-id: d807557f5e44279c49ab701b7a797253ef1f0717
Summary: While testing something for another change, I came across this overlooked typo.
Reviewed By: wez
Differential Revision: D22894060
fbshipit-source-id: 8aa48ef5da714650c974adcf8a34a542fdd4ed9e
Summary:
Avoid some overhead and complexity by storing BufVec as a
unique_ptr<IOBuf>. The complexity can be reintroduced if we ever find
FUSE splice support to be a performance win for us.
Reviewed By: kmancini
Differential Revision: D22710795
fbshipit-source-id: e58eedc0fb5cea9e9743ccd20d3e4e2b7cc5d198
Summary:
Previously we log a process to Scuba when it does 2000 (fetchThreshold_) fetchs, but then in Scuba all processes have fetch_count = 2000. In order to see how many fetches a process really did approximately, we log the same process to Scuba every time it does 2000 more fetches.
Note: this change could make the total count of fetch-heavy events in Scuba inaccurate, as we log the same process more than once. So when users want to see how many fetch-heavy events happened, instead of setting "type = fetch_heavy", they should set exactly "fetch_count = 2000".
Reviewed By: chadaustin
Differential Revision: D22867679
fbshipit-source-id: ae3c768a8d3b03628db6a77263e715303a814e3d
Summary: Since local store compaction is not a hard requirement for graceful restart, make this issue non blocking. We've seen some users fail restarts because they had compaction issues due to lack of space on their device. If we fail during the compaction stage, we should continue the restart anyway. This is also because there is a chance that the local store will clear columns that are no longer in use.
Reviewed By: chadaustin
Differential Revision: D22828433
fbshipit-source-id: 9a2aaec64e77c2d00089834fda8f8cffda472735
Summary:
TL:DR:
A codemod did something a bit unclean, so they added a lint. This will keep bugging us if we make changes here, so let's satisfy the linter.
More info:
`x.y_ref() = ...` and `*x.y_ref() = ...` are pretty much the same except `*x.y_ref() = ...` can throw for optional fields.
A codemod added a bunch of `*x.y_ref() = ...`, but after they didn't want people to copy paste this for optional fields so they added a lint that pops up on non optional fields too :(
https://fb.workplace.com/groups/thriftusers/permalink/509303206445763/
Reviewed By: chadaustin
Differential Revision: D22823686
fbshipit-source-id: b3b1b8a3b6b1f1245176be19c961476e4554a8e5
Summary:
Previously, fetch heavy event's cmdline was delimited by '\x00' when logged to Scuba. (for example: `grep--color=auto-rtest.`)
Now we replace \x00 with a space, so command name and args will be separated by space. ( `grep --color=auto -r test .` )
Reviewed By: kmancini
Differential Revision: D22772868
fbshipit-source-id: 4ab42e78c7bc786767eee3413b9586739a12e8ac
Summary:
This helps in understanding what's going on when some files disappear and/or
aren't flushed properly.
Reviewed By: fanzeyi
Differential Revision: D22833201
fbshipit-source-id: 09beb5796cb40c0a93107ee6a3a3497abb2578f0
Summary:
The tilde got dropped as part of the changes in D22672240 (be3683b1d4)
(an easy mistake to make!) and that renders this function less
useful.
Thankfully the caps display isn't a critical function; just for
some diagnostic printing.
Reviewed By: chadaustin
Differential Revision: D22847590
fbshipit-source-id: 716d7c7bd674260687fbc09e3dc94538359f98b3
Summary:
`log()` passes fsck bars to standard output, but it will also print the same message to the log with level DBG2. (example below)
```V0713 07:05:45.971511 3510654 StartupLogger.cpp:96] [====================>] 100%: fsck on /home/ailinzhang/eden-state/clients/dev-fbsource6/local
```
Since we don't want the log file to be messed up with fsck bars, we use `logVerbose()` with level DBG7.
Reviewed By: kmancini
Differential Revision: D22727965
fbshipit-source-id: 0700503af511030df2abbca4ad2fa1540995e919
Summary:
Unfortunately, using sqlite causes `edenfsctl prefetch` to take several orders
of magnitude more time than with rocksdb, tuning sqlite to be faster (at the
expense of reliability) still doesn't come closer to rocksdb. From profiling,
this is due to sqlite only allowing a single thread to read/write onto it
serializing all the work that EdenFS is doing.
A rework on the storage will be necessary to be able to get both good
performance, and reliability but that's a long term project, for now, make
Windows use rocksdb by default.
Reviewed By: wez
Differential Revision: D22819084
fbshipit-source-id: 62f397858ed547da30ef8a6346b767461dc53493
Summary:
As opposed to FUSE, ProjectedFS sends notifications for file/directory creation
after the fact, and for directory that means these will be visible on disk before
EdenFS may be aware of it. While EdenFS usually process it quickly, a heavily
multi-threaded application that tries to concurrently create a directory
hierarchy may end up sending notifications to EdenFS in a somewhat out of order
fashion.
Since this should be a very rare occurence, we make this a very slow path by
being optimistic and calling `getInode` first, and then only if that fails, we
aggressively create all the parent directories. During a buck build of ~1k
jobs, this happened only 3 times.
If we fully think this through, this change doesn't fully fix the race, as a
similar race can now happen when a create and remove/rename operations are
concurrent. However, a client performing these operations concurrently is
either aware that this is racy and should handle these properly, or is most
likely buggy. Both of these should significantly reduce the likelyhod of this
happening, thus, I'm leaving this unfixed for now.
To better understand how frequently this happens, I've added a stat counter.
For now, these aren't published to ODS, but this will be tackled later.
Reviewed By: wez
Differential Revision: D22783484
fbshipit-source-id: ea3aafc2f77b65d3967f697f68114921d5909137
Summary:
To avoid mistakes when refactoring, static_assert that the template
variant of sendReply and make_iovec are only used with POD types.
Reviewed By: genevievehelsel
Differential Revision: D22710742
fbshipit-source-id: 4557761f3946fe8969ce31a42502f64cc3298e1d
Summary:
Previously, we kept a list of weak pointers to RequestData instances
to track live FUSE calls, but the only thing we need is the FUSE
request header, so directly track that instead.
Reviewed By: kmancini
Differential Revision: D22710716
fbshipit-source-id: 3820acb314ac2db85b86de128fd082bc4871d9c6
Summary:
Now that we are explicit about whether the kernel caches must be
invalidated, we can remove a use of folly::RequestContext.
Reviewed By: kmancini
Differential Revision: D22710518
fbshipit-source-id: 4bd5267bf5dd3135adf33e4f4fa1ea2649816564
Summary:
Avoid the cost of dynamically querying whether we are in a FUSE
request handler or not by passing a flag.
Reviewed By: kmancini
Differential Revision: D22710480
fbshipit-source-id: 010bb8efee8074441aa20aab0eb12277452c5252
Summary:
Avoid the cost of dynamically querying whether we are in a FUSE
request handler or not by passing a flag.
Reviewed By: kmancini
Differential Revision: D22710452
fbshipit-source-id: 818035b72b793fa895147d9df3bb668d5b9c55f3
Summary:
Avoid the cost of dynamically querying whether we are in a FUSE
request handler or not by passing a flag.
Reviewed By: kmancini
Differential Revision: D22710422
fbshipit-source-id: 65b0737ad5f8ca74d12f2c657691d3751df4aa54
Summary:
Avoid the cost of dynamically querying whether we are in a FUSE
request handler or not by passing a flag.
Reviewed By: genevievehelsel
Differential Revision: D22710397
fbshipit-source-id: 7c62f45dfc227416c91070842a349b9d0c626cba
Summary:
We are unifying C++ APIs for accessing optional and unqualified fields:
https://fb.workplace.com/groups/1730279463893632/permalink/2541675446087359/.
This diff migrates code from accessing data members generated from unqualified
Thrift fields directly to the `field_ref` API, i.e. replacing
```
thrift_obj.field
```
with
```
*thrift_obj.field_ref()
```
The `_ref` suffixes will be removed in the future once data members are private
and names can be reclaimed.
The output of this codemod has been reviewed in D20039637.
The new API is documented in
https://our.intern.facebook.com/intern/wiki/Thrift/FieldAccess/.
drop-conflicts
Reviewed By: iahs
Differential Revision: D22764126
fbshipit-source-id: 67b1bc6d4a9135f594d78325cee8a194255bdcb8
Summary: This is unused, and won't ever be used.
Reviewed By: genevievehelsel
Differential Revision: D22744352
fbshipit-source-id: 9d20db608f972288eaf33e3ea0a79ffe5e13e03e
Summary: These were only used in tests, no need to keep it.
Reviewed By: chadaustin
Differential Revision: D22744353
fbshipit-source-id: 57596d641ab85f15e8c945327d7849a64aa73ef8
Summary:
Both unices and Windows needs to invalidate the cache in the same place, let's
avoid ifdef and consolidate the function name to clean things up a bit.
Reviewed By: chadaustin
Differential Revision: D22741709
fbshipit-source-id: 04060c0080eff9840abd22747ea48404fa50fd86
Summary:
We're experimenting with enabling NO_OPEN support in our internal
build of the osxfuse kext. This commit includes the relevant capability bits
for the kernel interface (which are compatible with the linux FUSE
implementation) as well as adjusts our FUSE client code to detect and use
those bits on macOS.
Reviewed By: xavierd
Differential Revision: D22744378
fbshipit-source-id: 21376439a85b0b0f5a71916dd1af618d9627695e
Summary:
Even though we never asked for a pre rename notification, ProjectedFS would
anyway send one to us, and since we're now failing on unrecognized
notification, that meant renames would always fail. Handle them and do nothing in them.
I'm not sure yet if we want to move the actual rename logic into the pre rename
callback, or keep it as is. The benefit of the pre rename code is that it can
fail the user request, while the current code will fail, but not prevent the
rename from happening.
Reviewed By: wez
Differential Revision: D22738388
fbshipit-source-id: 487e1f90b503bc59cff7315dd38d2a3039552eaf
Summary: This will make it easier to support PrjFS async programming model.
Reviewed By: chadaustin
Differential Revision: D22738408
fbshipit-source-id: 2e2e4f6f2f718b0226cf9fab66589c50b6db49db
Summary:
The pattern when using it is to catch the exception, then call the function,
which then would re-throw it and catch the subtyped exception. We can avoid a
re-throw entirely by having the function take the exception that was thrown in
the first place and dynamic cast it.
Reviewed By: chadaustin
Differential Revision: D22736772
fbshipit-source-id: 0efa3134bccf3ba8bdcd51d67e03c7ee4483a99f
Summary:
We're still reading the entire file at a time, but this paves the way to not do
that.
While the change looks big, a lot of it is just moving code around. The main
gist of it is removing EdenMount::readFile and writing the proper future
combinator in EdenDispatcher::getFileData.
Reviewed By: wez
Differential Revision: D22361748
fbshipit-source-id: 6391a29d25a4c9e61b91952c40c21ad52e728c8b
Summary:
This would enable the enumeration callback to use ProjectedFS asynchronous
completion.
Reviewed By: chadaustin
Differential Revision: D22361747
fbshipit-source-id: b2d31533ee5128e9dd3da7f91d5225331cf8e926
Summary:
We've had a long latent bug where for some reason a file that should be in the
repo isn't visible on the filesystem. Turning on logging and running `ls` shows
that EdenFS tells ProjectedFS about the file, but it's somehow not shown. The
reason being that ProjectedFS keeps track of all the removed files by adding a
tombstone for these, and will not list them until EdenFS removes this
tombstone.
Looking at the various places where we invalidate FUSE's cache, but not
ProjectedFS, only one stands out, and it has to do with files not being present
in the working copy, but added in the update destination. Adding the right
flush there appear to solve the simple repro listed below.
I'll remove all the ifdef around flushing in a later diff to avoid this issue
from re-appearing again.
Reviewed By: fanzeyi
Differential Revision: D22739916
fbshipit-source-id: 3a4fbc825cd21b36cbd2616882fd50e3d9741f63
Summary:
This adds logging for data fetches that come from the thrift globfiles call to
help debug the cause of unexpected data fetches. (See D22448048 for more
motivation)
Reviewed By: genevievehelsel
Differential Revision: D22489512
fbshipit-source-id: 040cf1277205d08ea864a1f30f8d3b25ee7c4508
Summary:
This adds logging for files fetched in prefetch like was aleady added for
blob and tree fetches.
This is needed to log the fetches caused by the glob files thrift call. The
purpose of this to help debug the cause of unexpected data fetches (See
D22448048 for more motivation).
Reviewed By: genevievehelsel
Differential Revision: D22561619
fbshipit-source-id: 5ae78b99fb0c7d863d8223b93492b0d0210ddf9e
Summary:
This adds logging for data fetches that come from the thrift getSHA1 call to
help debug the cause of unexpected data fetches. (See D22448048 for more
motivation)
Reviewed By: genevievehelsel
Differential Revision: D22489514
fbshipit-source-id: eb2d82c206af857cc79439d2854d682641292db8
Summary:
This adds logging for data fetches that come from the thrift getFileInformation call to
help debug the cause of unexpected data fetches. (See D22448048 for more
motivation)
Reviewed By: genevievehelsel
Differential Revision: D22489513
fbshipit-source-id: ab6283476d05b06b9f9e37c6b4fd81c1282046ff
Summary:
This adds logging for data fetches that come from the thrift checkout call to
help debug the cause of unexpected data fetches. (See D22448048 for more
motivation)
Reviewed By: chadaustin
Differential Revision: D22489504
fbshipit-source-id: 3b732a1e5627c2130f561ec0138a1df270e1925d
Summary:
We have seen that some of the unexpected data fetches do not originate from
FUSE. This adds parity to the logging for data fetches that come from the thrift
interface. Adding this logging improves the overall observability of eden, and
will help us debug the cause of unexpected data fetching.
This introduces plumbing to allow logging data fetches that originate from
thrift requests.
Reviewed By: chadaustin
Differential Revision: D22448048
fbshipit-source-id: a39dde72467c4922c07c569c14fb499341d40258
Summary:
What does this do: allocate a ThriftLogHelper and ensure it lives for the
duration of its coresponding thrift request.
Why: By ensuring that the log helper's lifetime matches that of the thrift
request, we simplify adding to the logger. We wont have to think about the
lifetime of each of the members of the logger, we simply can stick them in the
log helper and let shared_ptr do its magic.
More specific why: This refactor helps make adding Thrift data fetch logging
cleaner (see following changes). We need to ensure the ObjectFetchContext
stays around for the duration of the thrift request since we pass it around
by reference. Sticking this in ThriftLogHelper avoids adding another disjoint
piece of plumbing and makes the code easier to maintain going forward.
Reviewed By: genevievehelsel
Differential Revision: D22632546
fbshipit-source-id: 1baa79419386947e52a386d89a65f032f1988622
Summary: This is unused, no need to keep it around.
Reviewed By: wez
Differential Revision: D22361749
fbshipit-source-id: 3c353776437b59c6c7735652f7eb1ce052215e11
Summary:
Instead of overloading the FileMetadata struct that is used for directory
listing, let's use one tailored for our use. This will enable more flexibility
when deciding what to provide ProjectedFS. For instance, we could send the
InodeNumber so we don't need to do expensive path resolution for every notification.
Reviewed By: wez
Differential Revision: D22361751
fbshipit-source-id: 4801be45d8afc3af51e0a9564d9acb0a8e32255a
Summary:
Futures are intended to be chained together and not synchronously waited one
after the other. While this may be one of the goal, it also paves the way to
enable ProjectedFS asynchronous notification handling.
While doing this, a bunch of code was moved from EdenMount.cpp to the
dispatcher itself, the rationale behind this is to follow what the unix
EdenDispacher with the long term plan to merge the 2 as much as possible.
Reviewed By: wez
Differential Revision: D22361750
fbshipit-source-id: fa679a8b94ff6f8b5a33782fdb6b129ab066c4d8
Summary:
Whenever EdenFS reads a file from the overlay, there is a risk of this
triggering a prjfs notification, especially when there is a discrepancy between
the actual state of the overlay, and what EdenFS thinks the overlay should look
like.
A simple example of this includes removing a materialized file in the overlay
when EdenFS isn't running, and running `hg status` when it's back up. The
status call will try to compute the sha1 of the file, and thus will call into
the `CreateFile` API, which will then trigger an EdenFS callback to be called.
This callback will then try to recursively acquire the inode lock, which is
already held by the sha1 computation code. And that's a textbook example of a
deadlock.
To avoid this in the first place, let's simply refuse recursive callbacks. The
sha1 code will either have to be modified to handle errors, or we need to scan
the overlay when mounting the repo.
Reviewed By: fanzeyi
Differential Revision: D22288431
fbshipit-source-id: 2b4b31eddf7debcde55c22dd704d198ec44e59b4
Summary:
At mount time, EdenFS will try to create the .eden/config file which will
indicate to the client that this is an EdenFS repo, this config also contains
the location of the socket to talk to EdenFS.
On unices, and while the .eden directory is slightly different, the content of
this directory is written onto the overlay, but on Windows, the overlay is the
repository itself. What this means is that creating this config file will end
up triggering a ProjFS callback, which can potentially lead to deadlocks if
we're not careful.
A future change will simply prevent these recursive callbacks from happening
and thus a solution needs to be found for the .eden/config file itself. Since
the file itself is tiny (about 400B), and should only be accessed once[0], the
simple approach to keep it in memory and special case it works perfectly.
[0]: Once a file was read fully by ProjFS, it's present in the overlay and
EdenFS will no longer be requested to provide it.
Reviewed By: chadaustin
Differential Revision: D22310734
fbshipit-source-id: 6b2dba2164496ebd251104d7875b51569be2471f
Summary: Previously, `BackingStore` and all its sub-classes' `getBlob` and `getTree` methods accepted both `ObjectFetchContext` and `ImportPriority` as arguments. Now, `ImportPriority` is removed because we can get the priority from `ObjectFetchContext `
Reviewed By: kmancini
Differential Revision: D22650629
fbshipit-source-id: e1b0c57a059f11504b28b2c17d698bb58f51e1ee
Summary:
check fetch count before `getPriority()` is used. If fetch count has exceeded `fetchThreshold_`, lower the priority by 1.
Note: this diff only guarantees that the `getPriority()` function is returning the lowered priority. How the returned value is used for scheduling is handled by `HgQueuedBackingStore`
Reviewed By: kmancini
Differential Revision: D22550640
fbshipit-source-id: c032f8f72ca658618ac118dfb3ad3dcae61e9735
Summary: Previously `getPriority()` always returned a fixed priority. Now that we want `ObjectFetchContext` to lower its priority, it is better to make it have its own copy of `ImportPriority`.
Reviewed By: kmancini
Differential Revision: D22550587
fbshipit-source-id: 029c797def477ae4533f66cfba146a3972cfb65d
Summary: This diff ensures that the value of ImportPriority is always positive and changes offset from 0 to half of the maximum offset to allow lowering offset in the future.
Reviewed By: kmancini
Differential Revision: D22550462
fbshipit-source-id: 69f45369359c7b2c39a0c6831e9b33982e26a16a
Summary:
within() schedules work on the singleton timekeeper thread which
sometimes scheduled a timer callback on the ServerExecutor while
TestMount was shutting down, causing an assertion failure. Since
TestMount's executor is deterministic, just make the entire test
deterministic.
Reviewed By: genevievehelsel
Differential Revision: D22709150
fbshipit-source-id: 0a1621df0a9934bb7944fc259f7f83d110b35e93
Summary: If the user specifies a mount, only clear for that mount, otherwise clear for all mounts.
Reviewed By: fanzeyi
Differential Revision: D22674210
fbshipit-source-id: 832c4fd37a63fdb44cb12844378b61d113731016
Summary: add a thrift call to clear `pidFetchCount_` in `ObjectStore` and call it in `eden debug gc_process_fetch`. Users might want this command to start a new recording of process fetch counts.
Reviewed By: kmancini
Differential Revision: D22583430
fbshipit-source-id: eba7d63b08da5134fd09b7512895aba06f6a7ca5
Summary:
In Windows 10 2004, the behavior of AF_UNIX socket changed slightly, a socket
on which a server is listening to now appears to be shown as a "reparse point"
on disk. This appears to confuse Python as `os.path.exists(socketpath)` no
longer returns `True` for a listened to socket, which means that edenfsctl
would not be able to communicate with EdenFS, even though it is up and running.
Windows still has this weird behavior where the socket is created on disk
during connect, but EdenFS knows about this and already retries several times
(D21755498 (2c5768404d)) to avoid the issue, it's therefore fine to remove this code.
Reviewed By: wez
Differential Revision: D22666081
fbshipit-source-id: e2eea3fdd7628354b8b187b76899be4e9e2fa17f
Summary:
lower the debug level for scs proxy hash logging so that this does not cause so
much noise in the logs of production eden
Reviewed By: chadaustin
Differential Revision: D22668574
fbshipit-source-id: 1a7c2a4706514c0ef0bb848424681eef9316d296
Summary:
We are unifying C++ APIs for accessing optional and unqualified fields:
https://fb.workplace.com/groups/1730279463893632/permalink/2541675446087359/.
This diff migrates code from accessing data members generated from unqualified
Thrift fields directly to the `field_ref` API, i.e. replacing
```
thrift_obj.field
```
with
```
*thrift_obj.field_ref()
```
The `_ref` suffixes will be removed in the future once data members are private
and names can be reclaimed.
The output of this codemod has been reviewed in D20039637.
The new API is documented in
https://our.intern.facebook.com/intern/wiki/Thrift/FieldAccess/.
drop-conflicts
Reviewed By: yfeldblum
Differential Revision: D22631599
fbshipit-source-id: 9bfcaeb636f34a32fd871c7cd6a2db4a7ace30bf
Summary: add `--mount` option to `eden debug processfetch` as users might want to know under which checkout they have run those commands.
Reviewed By: kmancini
Differential Revision: D22485225
fbshipit-source-id: df321e6c30064c17abc62584863119b362b2c296
Summary: Eliminate extra blank line in tabulate
Reviewed By: kmancini
Differential Revision: D22477405
fbshipit-source-id: b867bfa5c971fcac0104563e866a314416f24fb0
Summary:
This diff added a sub-command `process_fetch` to `eden debug`, which lists eden processes sorted by their fetch counts.
New argument options will be added in following diffs.
Reviewed By: fanzeyi
Differential Revision: D22456147
fbshipit-source-id: 75f94ec0ad03d59af1a5bf31c37bdf56de70241b
Summary:
To help investigate a Linux kernel performance dropoff where the
kernel falsely thinks the EdenFS FUSE mount can't handle dirty page
writes at a high rate, add a 4k random writes benchmark.
Reviewed By: simpkins
Differential Revision: D21328771
fbshipit-source-id: c9977bd2e291d01e92631094aba3d8d807ec62da
Summary:
Replace calls to Future<T>::getTry() with Future<T>::result().
This change is behaviour-neutral. It enables us to make the behavior of getTry
match expectations and be blocking, like get, and r-value qualfiied, like
SemiFuture<T>::getTry().
Reviewed By: yfeldblum
Differential Revision: D22527935
fbshipit-source-id: 7fa634375b71c13ee58cb699e5ac8dcd37fe9c30
Summary:
Deprecate server-side minCompressBytes.
Background:
The minCompressBytes on ThriftServer is another knob to control whether or not to compress payloads. The reason to deprecate it is twofold:
(1) to provide a single place for compression control.
(2) 99.99% of services only copy-paste the default value (200) anyways.
Since rocket transport will be the default transport for all future thrift RPC, D21702725 introduced the ability for client to pass CompressionConfig to server (in rocket) which informs server both codec and size limit. This way the compression is enabled and controlled only from the client-side.
Reviewed By: iahs
Differential Revision: D22354858
fbshipit-source-id: dd920d4d38969f8ee7894b1aaafbe71c0e8b8d86
Summary:
This function is only used in EdenDispatcher, and all of it can be achieved by
simply calling InodeBase::stat method to fetch the file size.
As a bonus, the code now uses future continuation, instead of individual calls
to `get`.
Reviewed By: chadaustin
Differential Revision: D22310733
fbshipit-source-id: c81be7dd114c3a9bebc6b1b5b0887ae3ade30602
Summary:
The WinStore was only used for 2 simple things: checking that a file is
present, and getting its blob. Since both of these are trivial to implement
with EdenMount, there is no reason for WinStore to exist.
Reviewed By: chadaustin
Differential Revision: D22288430
fbshipit-source-id: 90567ac39b5124039becd3599766fcd0cde3e9aa
Summary:
While there isn't a direct equivalent to the unix RSS and VSZ, the working set
size and the page file usage can be used instead, they roughly have the same
meaning and that's probably good enough.
I've also moved the private bytes into the MemoryStats as it's a memory stat
too.
Reviewed By: chadaustin
Differential Revision: D22081006
fbshipit-source-id: d597d523eee21e7651a14e60b1fd6dc152a61185
Summary:
Thrift setter API is deprecated since it doesn't bring any value over direct assignment. Removing it can reduce build-time and make our codebase more consistent.
If result of `s.set_foo(bar)` is unused, this diff replaces
s.set_foo(bar);
with
s.foo_ref() = bar;
Otherwise, it replaces
s.set_foo(bar)
with
s.foo_ref().emplace(bar)
Reviewed By: chadaustin
Differential Revision: D22487354
fbshipit-source-id: d2f77c1410dc9b83805cb12f818f01eb939d0e29
Summary:
Replace calls to Future<T>::getTry() with Future<T>::result().
This change is behaviour-neutral. It enables us to make the behavior of getTry
match expectations and be blocking, like get, and r-value qualfiied, like
SemiFuture<T>::getTry().
Reviewed By: chadaustin
Differential Revision: D22510034
fbshipit-source-id: bd45cc6a404293089371654829a63c81b1c706aa
Summary:
AsyncSocket::newSocket will soon return a unique_ptr. In preparation,
explicitly update the call sites to move instead of copy the socket
when possible.
Reviewed By: simpkins
Differential Revision: D22402156
fbshipit-source-id: 52580826fb70fea915cf124d264bbe13bd56484e
Summary:
This extends the metadata importer to work on corp.
Prefetching metadata for the entries in a tree when we fetch it saves us
an extra round trip to the server to fetch a blob when only the metadata
for that blob is fetched. This is particularly important on corp where the
latencies to the server are larger.
Requesting metadata with out the blob itself happens often in the parsing
phase of buck commands, thus this metadata prefetch should improve
the build times, and other work flows that rely heavily on buck queries.
Reviewed By: chadaustin
Differential Revision: D21698728
fbshipit-source-id: 4072be23f2fa7df33cf46879e8c1d8ddd6c316ba
Summary:
Buck uses the content SHA-1 to identify each of the source files for a target.
During the parsing phase it needs these SHAs, though the content of the
files is not yet needed, and may never be needed if the file has already
been built and is in the buck cache.
Currently, if we do not already have metadata cached for a file when
requested we fetch the contents of the file, and compute the hash.
We want to avoid this.
Eventually this data will be available from the Mononoke EdenAPI server,
but for now we want a temporary solution to unblock the Buck team, and
ship benefits early.
Reviewed By: chadaustin
Differential Revision: D21820913
fbshipit-source-id: 56a7e32519f0fb04881518306d94aaed33527fd9
Summary:
Prefetching metadata for the entries in a tree when we fetch it saves us
an extra round trip to the server to fetch a blob when only the metadata
for that blob is fetched. (This can happen often while parsing targets in
builds)
We will to prefetch the metadata for each of the entries in a tree when
we fetch the tree and store the metadata for each entry under that
entries id (to make looking up the entry metadata by its id quick)
However, we also don't want to unnecessarily fetch data from
the server if we already done so.
To accomplish this we will also store the metadata for each entry under the tree
id in the local store. This will: 1) allow us to check if we have already fetched
the metadata from the server when we are fetching a tree (we only have the
tree id easily available here to storing the metadata under the tree id makes
it much easier/less expensive to do this check). 2) allow us to refil the
metadata for each entry stored under that entries blob id if it has been cleared
from the local store (this may happen is the local store is gets large and gets
partially cleaned to reclaim space).
This implements the method to store tree metadata for all entries under the tree
id and under the blob id for each entry.
Reviewed By: chadaustin
Differential Revision: D22239173
fbshipit-source-id: d4e0ffd642ce0b4034188cfc4eeaf2ea05f54e77
Summary:
Prefetching metadata for the entries in a tree when we fetch it saves us
an extra round trip to the server to fetch a blob when only the metadata
for that blob is fetched. (This can happen often while parsing targets in
builds)
This implements a custom metadata fetcher to fetch this data when we
fetch a tree from the server.
Reviewed By: chadaustin
Differential Revision: D22086639
fbshipit-source-id: 5fe31d375bf6f7376eb67496d553d6b4540fc0c9
Summary:
This will allow adding custom MetadataImporters in different eden builds.
DefaultMetadataImporter provides a no-op version of the interface to be
used by default.
Reviewed By: chadaustin
Differential Revision: D21960834
fbshipit-source-id: aec8a3627ab1223f74466b92a0ebe3290b67b7ed
Summary:
Previously the BackingStore kept a raw pointer to the LocalStore. To do this we relied on EdenServer ensuring the lifetime of the LocalStore exceeds that of the BackingStore.
This makes the LocalStore pointer a shared pointer to explictly make sure that the LocalStores lifetime matches the BackingStores lifetime.
Reviewed By: chadaustin
Differential Revision: D22394597
fbshipit-source-id: c81cb26c6fc8f834bc46d8576ced06ba6a96ac2c
Summary:
This introduces a class to manipulate the metadata for all the entries in a
tree. This adds serialization and deserialization to this class so that it can
be written to the local store.
Why do we need this? We need some way to easily check when we have already
fetched metadata for a tree and do not need to refetch this from the server to
avoid expensive network requests. Later diffs add functionally to store the metadata
for tree entries in the local store under the tree hash using this class.
Reviewed By: chadaustin
Differential Revision: D21959015
fbshipit-source-id: 0c0e8750737f3076c1f9604d0319cab7f2658656
Summary:
In following diffs we will use scs to prefetch meta-data for files, so that this data
will be available with out fetching the file content (which will improve build times
on eden).
This builds up the proxy hash index that serves as a conversion between eden
specific identifiers and commit and path which we will use to index into scs.
Reviewed By: chadaustin
Differential Revision: D21820909
fbshipit-source-id: 17891f6772f49c7c183061d7a4df2fe0a3be9d25
Summary:
In following diffs we will use scs to prefetch meta-data for files, so that this data
will be available with out fetching the file content (which will improve build times
on eden).
SCS indexes trees by an scs specific hash (blake2 content hash) or by the commit
hash and path. Since this is different from the eden hashes and mercurial
hashes, we need another index to go between the current ids we have in eden
and identifiers for scs.
This introduces a proxy hash that serves as this conversion. Because we have
commit hashes around in eden right now, this is an easier route to indexing
into scs currently.
Reviewed By: chadaustin
Differential Revision: D21237648
fbshipit-source-id: 79115ac034a5f062ae879713cd2c1a17f348c725
Summary:
There was some missed usage of `Path.resolve`. This diff should cover it all.
```
cli $ rg -F ".resolve"
main.py
967: uid = self.resolve_uid(args.uid)
968: gid = self.resolve_gid(args.gid)
util.py
622: `Path.resolve`. This is a helper method to work around that by using
628: return path.resolve(strict=strict)
```
Reviewed By: chadaustin
Differential Revision: D22459188
fbshipit-source-id: c2a1b132f752cc399ebf34723f26123559939f2a
Summary:
proxy_import_helper.py exists for compatibility with older EdenFS
builds. None of those builds are running anymore, so remove it.
Reviewed By: genevievehelsel
Differential Revision: D22451196
fbshipit-source-id: 4d258b3fafe13bb67bd11259f5d1193a7e5575e6
Summary: This diff defines `Overlaychecker::ProgressCallback` to replace repetitive function type declaration.
Reviewed By: genevievehelsel
Differential Revision: D22243160
fbshipit-source-id: ea05e451817a760b5266879b956eaea48dc8d85e
Summary:
From the Rocks DB documentation:
> When opening a DB in a read-write mode, you need to specify all Column
Families that currently exist in a DB. If that's not the case, DB::Open call
will return Status::InvalidArgument()
This can cause problems for us in a couple of situations:
- When we need to rollback from an eden version where we added a column to
our configuration for RocksDB
- When we delete a column from our configuration for RocksDB
To make sure we do not encounter this error we need to make sure that we still
open all the columns existing in the database, even if they are not in our
configured list of family columns.
Reviewed By: wez
Differential Revision: D22425310
fbshipit-source-id: 9822b22cfedf4633f65bbed96f95a546dd3614f4
Summary:
Currently when we are resolving the full command line for a client pid, we only
read the first 256 bytes of the command.
This means that some commands will be truncated, this has come up in some
of our recently added logs. This ups the buffer size so that we can
hopefully get the full command line.
The longer term solution would be to implement the something fancier mentioned
in the comment in the code copied below, but also has drawbacks as mentioned.
> // Could do something fancy if the entire buffer is filled, but it's better
// if this code does as few syscalls as possible, so just truncate the
// result
Reviewed By: wez
Differential Revision: D22436219
fbshipit-source-id: 80a9aecfe148aa3e333ca480c6a8cb8b9c5c86f2
Summary:
Rather than dynamically allocating an event loop in the systemd async
code, make all the corresponding functions async, so the caller is
responsible for threading an event loop down.
Reviewed By: genevievehelsel
Differential Revision: D21894106
fbshipit-source-id: 398c769c30c85a3bb210dbc209f34f9f7336996c
Summary: I'd eventually like to use this in the edenfs_monitor, so moving it to `proc_utils` for sharibility.
Reviewed By: chadaustin
Differential Revision: D21998763
fbshipit-source-id: 052e78fb8e58515f98eb465b8041fd0e621fc9da
Summary: I'd eventually like to use this in the edenfs_monitor, so I'm adding this to `proc_utils` for future ease of use.
Reviewed By: chadaustin
Differential Revision: D21987390
fbshipit-source-id: 076672b44311c2a1e0cac934c0674a18a87649af
Summary: On macOS, "Icon?" (aka "Icon\r") is a sometimes added. This file is weird to ignore, and should be ignored using "Icon\r\r" or "Icon[/r]". This won't be hidden with "Icon\r" or "Icon\r"
Reviewed By: chadaustin
Differential Revision: D22050682
fbshipit-source-id: 51d7d4c2414a07b959120455ae991d2425c1ea4d
Summary: [edenfs] Some minor modifications to verbiage in documentation
Reviewed By: chadaustin
Differential Revision: D22394129
fbshipit-source-id: 2d662e56d621fd5e5d5ba6de284ca3d08f8bd4e5
Summary:
This is much better than having `ObjectFetchContext` itself owns a copy of `ImportPriority`. We can actually customize how different fetch context manages these priority.
We set all FUSE requests to a higher priority and prefetch requests to a lower priority
Reviewed By: xavierd
Differential Revision: D22342802
fbshipit-source-id: b9c1d0f2ddbc7a5e5d619bc2c2222e5df0e702af
Summary:
This commit makes prefetch requests derived in `TreeInode` assigned to lower priority.
Since prefetch is not running under the original request's context, we create a new `ObjectFetchContext` specifically for this case, and attach it to `prefetchLease` since it is guaranteed to be alive during the span of prefetch.
Reviewed By: chadaustin
Differential Revision: D21872719
fbshipit-source-id: d1495b06031980d7d1c21ecf2a0b47e83fc9672d
Summary:
This commit adds `ImportPriority` to `ObjectFetchContext`. By doing so we can tweak priority for a request at different stage.
This commit also provides a default implementation for the virtual methods in `ObjectFetchContext` so we can create one to carry `ImportPriority` in some specific cases.
Reviewed By: chadaustin
Differential Revision: D21872718
fbshipit-source-id: 6e8cfd84959b368e6fe69fda2baf0debf7a88295
Summary: This commit makes `readdir()` to correct send its `RequestContext` to `EdenDispatcher` method so underlying code can know the current request being processed is from FUSE.
Reviewed By: xavierd
Differential Revision: D21821949
fbshipit-source-id: f41ba912fedbfc040e3c9267aad25e7f33f8e912
Summary: This diff made fetch threshold configurable, so we can change it later as repository size grows.
Reviewed By: fanzeyi
Differential Revision: D22337850
fbshipit-source-id: 4b46420cb4e7164a3f1080279d67fa5f90549cd8
Summary: This diff updated `ObjectStore` to send a `FetchHeavy` event to Scuba when the number of fetching requests of a process has reached 2000.
Reviewed By: fanzeyi
Differential Revision: D22292104
fbshipit-source-id: b7ac48412868216ea960c8681a5fb71c587952bc
Summary:
Hardlinks aren't supported on EdenFS, let's not silently allow them since this
can cause unintended behavior. It's possible that this might break some
workflows, but it's better to be strict about this.
Reviewed By: genevievehelsel
Differential Revision: D22250849
fbshipit-source-id: 9f90e6a8e9c9715bb4b925c3d7f952e965683081
Summary:
Errors are usually a sign that we're doing something wrong. Ignoring them can
thus lead to weird behavior, hence let's not ignore them. This may have the
potential to breaking user applications if they were doing something we're not
handling properly, but it was likely that EdenFS was broken in a subtle way. We
should be able to better understand what's wrong and fix it properly then.
Reviewed By: genevievehelsel
Differential Revision: D22250852
fbshipit-source-id: 91daa60065b5621542aee1ca3207d1d7c54f8639
Summary:
When files are moved in and out the repo, the paths being passed in might be
empty. For code clarity, let's treat these as file creation and removal.
We also do not need to actually tell ProjectedFS about removed files, since
ProjectedFS is actually telling us that the file is moved/removed.
Reviewed By: fanzeyi
Differential Revision: D22250851
fbshipit-source-id: a678e58eb9c36d4dcbf1bb41d56b38fef8943284
Summary:
If we don't wait on these futures, they might never execute, which may lead to
EdenFS not being aware of files no longer present in the overlay.
For now, just add .get() to these futures, but really, we need to make these be
futures themselves and use continuation style code.
Reviewed By: fanzeyi
Differential Revision: D22326570
fbshipit-source-id: 7584981bec284f35f8d2689a3a5dafd1a111a3d8
Summary:
An assertion error is raised if `eden doctor` is in the middle of a merge. This is because we enter a specific "if" condition in the case that mercurial has two parent commits, and EdenFS only ever tracks `p0`, so EdenFS simply sets `p1` to the null commit in `_select_new_parents()`. Specifically, this is in the case in which both `_old_dirstate_parents` and `_old_snapshot` are not None.
Because `_old_dirstate_parents` has `p1` set to nonnull, and Eden thinks it is null , the check `self._new_parents != self._old_dirstate_parents` would be `True` even though there was actually no error.
Reviewed By: chadaustin
Differential Revision: D22048525
fbshipit-source-id: 9a19cc092e2bd80db0e01fb38533a1007640bee6
Summary: None of this code is used, just remove it.
Reviewed By: genevievehelsel
Differential Revision: D22083565
fbshipit-source-id: ad24a0e8ea04797e146ebb9b99c125539197cb89
Summary:
Older versions of EdenFS do not return the `fetchCountsByPid` field in the
`getAccessCounts()`.
The Python thrift client code returns this as `None` instead of as an empty
dictionary. This behavior arguably seems like a bug in the thrift code, since
the field is not marked optional. However, updating the thrift behavior would
have much wider implications for other projects. Additionally it's probably
not worth putting a lot of effort in to the older "py" thrift generator code.
Update the `edenfsctl` code to explicitly use an empty dictionary if the value
received from the thrift call is `None`
Reviewed By: fanzeyi
Differential Revision: D22302992
fbshipit-source-id: eced35a19d86e34174f73e27fdc61f1e2ba6a57f
Summary: The method name is symlink, not rmdir.
Reviewed By: genevievehelsel
Differential Revision: D22291297
fbshipit-source-id: 5dc37b053e06c965fd47df79990fc40adc097f87
Summary:
While suppressing notification should in theory be faster than getting
notification, it however causes a couple of issues due to the fact that EdenFS
is simply not creating inodes for any of the files in these directories.
Renaming a file in and out of these directories would for instance fail, due to
the inode not being present.
Since these directories are very low traffic overall and on Linux/macOS these
are not treated specifically (well, except for .eden, but that's a different
story), let's do the same on Windows.
Reviewed By: chadaustin
Differential Revision: D22250850
fbshipit-source-id: c13ed29faedc33c98b1a30227e44afc3f2c84c89
Summary:
This was causing `hg mv` to fail due to trying to hash a unicode path, but
Python3 refuses to hash anything but bytes.
Reviewed By: DurhamG
Differential Revision: D22235561
fbshipit-source-id: 3eb80b8e02d442a4036ab7be7ea5c139bd24ff5e
Summary:
On Windows, paths are case insensitive (but the filesystem is case preserving),
and thus `open("FILE.TXT")` and `open("file.txt")` refer to the same file. When
that file is not materialized and its parent directory isn't yet enumerated,
PrjFS will call the PRJ_GET_PLACEHOLDER_INFO_CB with the file name passed in to
the `open` call. In this callback, if the passed in name refers to a valid
file, it needs to call PrjWritePlaceholderInfo to populate the directory entry.
Here is what the documentation for that function states:
"For example, if the PRJ_GET_PLACEHOLDER_INFO_CB callback specifies
dir1\dir1\FILE.TXT in callbackData->FilePathName, and the provider's backing
store contains a file called File.txt in the dir1\dir2 directory, and
PrjFileNameCompare returns 0 when comparing the names FILE.TXT and
File.txt, then the provider specifies dir1\dir2\File.txt as the value of
this parameter."
While the documentation doesn't state how that name is used internally, we can
infer (and test) that the returned case will be used as the canonical
representation of that file, ie: the one that a directory listing will see.
Since the PathMap code already does a case insensitive search, we just need to
make sure to use what it returns instead of re-using the name used for the search.
The only caveat to all of this is the original comment that describe that
`metadata.name` can't be used as it causes crashes. From what I can tell, this
was written in later 2018, and I believe is no longer relevant: the
`metadata.name` field was simply not populated.
Reviewed By: wez
Differential Revision: D21799627
fbshipit-source-id: aee877cc2d5f057944fcd39b1d59f0e97de6315c
Summary:
updated `eden top` to:
- obtain PID-fetchCounts data from the updated -`getAccessCounts` thrift call in the previous diff
- display that data in a new column `FUSE FETCH`
Reviewed By: kmancini
Differential Revision: D22101430
fbshipit-source-id: 6584e71ce3a4629c73469607ca0a4c6ffd63e46f
Summary:
This diff updates `getAccessCounts` to
- obtain the PID-fetchCount map data added in the previous diff
- put that data into its `result`
Reviewed By: kmancini
Differential Revision: D22101232
fbshipit-source-id: 1d41715339d418b03c17f6c93a7a497b432973ae
Summary:
Both optional and pid_t weren't found and the right includes needed to be
provided. On Windows, the ProcessNameCache isn't compiled (yet), and since it
looks like the process name is optional in the BackingStoreLogger, let's not
provide it for now.
Reviewed By: fanzeyi
Differential Revision: D22215581
fbshipit-source-id: 31a7e7be62cd3d14108dc437d3dfabfb9e62f8d5
Summary:
On Windows, `pathlib.Path.resolve` has an use of uninitialized memory bug in Python 3.6.2. Lego Windows has this version of Python 3 installed.
When `eden prefetch` runs, it will first attempt to read the config file at the root of the repository (`$REPOROOT/.eden/config`). Because of this bug, when we normalize the path in edenfsctl, it will get some random bytes as a result. This subsequently causes `toml.read` to fail as it is unable to read a path containing NUL byte.
As you can see, this is the same exception as we see on Windows:
```
In [1]: open("test\x00")
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-1-57764b20d660> in <module>
----> 1 open("test\x00")
ValueError: embedded null byte
```
Reviewed By: xavierd
Differential Revision: D22177997
fbshipit-source-id: ab2565c8946d809bc15bc1597b749fb5e9440ca0
Summary:
`eden top` often adds extra quotes to the end of the commands in the process
table, this removes those pesky quotes
Reviewed By: genevievehelsel
Differential Revision: D21440316
fbshipit-source-id: f1200a28a5345691fcce517526d119f44e5993d0
Summary:
Eden can sometimes unexpectedly fetch files from the server, and we want
to know why this is happening. This adds logging for the source of
data fetching in edens backing store to help obviate why these fetches
are happening.
This temporarily adds the logging in the HgQueuedBacking store to get a naive
version of logging rolled out sooner. Follow up changes will move this logging
closer to the data fetching itself if possible (in HgDatapackStore and HgImporter).
Reviewed By: chadaustin
Differential Revision: D22012572
fbshipit-source-id: b1b012ce4ee133fbacecd586b7365c3c5a5386df
Summary:
We have seen that eden will unexpectedly fetch data, we want to know why.
This adds the plumbing to interact with edens current logging to be able to
log when eden fetches data from the server and what caused eden to do this
fetch. Later changes will use the classes created here to log the cause of data
fetches.
Reviewed By: chadaustin
Differential Revision: D22051013
fbshipit-source-id: 27d377d7057e66f3e7a304cd7004f8aa44f8ba62
Summary:
Recently the server team added an un-used directory to test that eden would not
fetch these as a test for the upcoming repo merge. They saw that these files
were fetched a non trivial number of times. We want to know why eden is causing
these fetches.
This adds the pid and interface through which the client is interacting with eden to
ObjectFetchContext for this purpose. This information will be logged in later
changes.
note: currently this is only for fetches through Fuse (thrift interface to follow).
Reviewed By: chadaustin
Differential Revision: D22050919
fbshipit-source-id: 49b93257a0e6d910f48b1e8ec6471527e056d22a
Summary:
This passes ObjectFetchContext into the backing store to prepare for adding
logging for the cause of server fetches.
In following changes I will add logging in the HgQueuedBackingStore.
Ultimately we will want to move this logging to be closer to the data fetching
(in HgDatapackStore and HgImporter), but I plan to temporarily add logging to
the HgQueuedBackingStore to simplify so that we can more quickly roll out.
Reviewed By: chadaustin
Differential Revision: D22022992
fbshipit-source-id: ccb428458cbf7a1e33aaf9be9d0d766c45acedb3
Summary:
In following changes I will be threading ObjectFetchContext into the backing
store importing process, since this will start to be used more outside of the
ObjectStore, I am moving this class into its own files.
Reviewed By: chadaustin
Differential Revision: D22022488
fbshipit-source-id: 1a291fea6e0fd56855936962363dfc9f6de8533d
Summary: In advance of peppering async everywhere, allow subcmd's run method to be async.
Reviewed By: genevievehelsel
Differential Revision: D21892187
fbshipit-source-id: f611faacf95649d8bb5588aeefc4546bd5f63984
Summary: This diff adds a PID-fetchCounts map to `ObjectStore` and makes `ObjectStore` update that map after every `didFetch`
Reviewed By: kmancini
Differential Revision: D22100413
fbshipit-source-id: 740342c7b4a453fe482344c2db9542381c3772e4
Summary:
On Windows, we can't rely on the .eden directory to not be present to decide if
the repository is mounted. Instead, we can just let edenfs decide and catch the
exception if it tells us that it's already mounted.
Reviewed By: fanzeyi
Differential Revision: D21934538
fbshipit-source-id: 60ed9f530456b627091f95f7387dc4b8f3a8dc5c
Summary:
All the unification of the mount code lead to this point, where supporting
unmount is merely just removing the various #ifdef.
Reviewed By: fanzeyi
Differential Revision: D21797078
fbshipit-source-id: 7602eb85e3fa276b72daebda36ae269ec38cde66
Summary:
On unmount, the code relies on the channel (fuse/prjfs) to set a promise, while
not strictly required on Windows as the interface is synchronous, this will
enable unmount to be the same between on all platforms.
Reviewed By: fanzeyi
Differential Revision: D21797081
fbshipit-source-id: c3eac448d2b13ef49db7a7d1f36bb084130ea63a
Summary:
Most of what fuseMount does can be shared between fuse and prjfs, so let's
rename it and start using it on Windows.
Reviewed By: fanzeyi
Differential Revision: D21797079
fbshipit-source-id: b92a36ecc702ee72df8ebdf9534add42bea9b97e
Summary:
When the scuba telemetry logger is unavailable, fall back on the
subprocess.
Reviewed By: genevievehelsel
Differential Revision: D21687855
fbshipit-source-id: c58a4bc1b34974add35d194c8aafdaac7ed47fdb
Summary:
On Windows (haven't verified on other platforms), ThreadLocalPtr don't appear
to be releasing resources when a thread die. This means that when the importer
thread dies, the actual importer (hg.real) would still run and use resources,
with no way of talking to it.
To fix this, let's manually reset it when the main thread function returns,
this forces the importer to be destroyed and therefore the various handles to
hg.real to be released, effectively terminating it.
I'm not sure if this is the proper fix, but delving into folly feels a bit
daunting. Keeping a TODO for later to go back to it and fix it properly in
folly.
Reviewed By: chadaustin
Differential Revision: D22012540
fbshipit-source-id: 99f994bb5128b38ccf8474031763b8a21055759a
Summary:
This diff makes `lookup` to use `RequestData` as `ObjectFetchContext` in `getattr` calls.
This will make sure we correctly record backing store fetches in `eden top`
Reviewed By: chadaustin
Differential Revision: D21780969
fbshipit-source-id: 468e2fadcebf4a00477bc5de434e6c658b99d1ce
Summary: This diff makes `InodeBase::stat()` to be able to accept `ObjectFetchContext` as a parameter.
Reviewed By: chadaustin
Differential Revision: D21780883
fbshipit-source-id: 9b1db2e2268cb98663bcf902ea61897da593ea05
Summary: This diff starts to use `RequestData` as `ObjectFetchContext` in our `read()` methods. This ensures EdenFS could track backing store fetches happened in FUSE requests.
Reviewed By: chadaustin
Differential Revision: D21792503
fbshipit-source-id: 9509a1bc8f28100a0dfe196e312c4785c7842345
Summary:
Previously we check if a request is a fuse request when we fetch anything from backing store, so we can collect number of fetches happened for each process in eden top.
This is creating a dependency from store to fuse, which is a little awkward. Instead, we could make `RequestData` a `ObjectFetchContext` and record the fetches when that happens.
Similarly in the future we should also have something equivalent in our Thrift layer.
Reviewed By: kmancini
Differential Revision: D21775919
fbshipit-source-id: 95056830ddbe7c999051c43e0d8eca9a67350904
Summary:
This diff updated the `eden du` command. Instead of printing
```Reclaim space from the LFS cache directory by running:
hg -R /data/users/akushner/eden-repos/opsfiles gc
Reclaim space from the LFS cache directory by running:
hg -R /data/users/akushner/eden-repos/megarepo_test gc
Reclaim space from the LFS cache directory by running:
hg -R /data/users/akushner/eden-repos/instagram-server_test gc
```
We now print "Reclaim.." only once and put available commands together
```
LFS cache detected in backing repo. To reclaim space from the LFS cache directory, run:
hg -R /data/users/akushner/eden-repos/opsfiles gc
hg -R /data/users/akushner/eden-repos/megarepo_test gc
hg -R /data/users/akushner/eden-repos/instagram-server_test gc
```
and did the same for working copy under backing repositories.
Reviewed By: fanzeyi
Differential Revision: D22038555
fbshipit-source-id: c19869db6f91e90a0627b3bb4b4f7e95142ae198
Summary: This diff fixed small problems in previous `eden du` summary printing and added a reminder of option `--clean` to users.
Reviewed By: fanzeyi
Differential Revision: D22038552
fbshipit-source-id: 7dbac858b74833e59d8d3f1c6fb1f5d8944935d0
Summary: This diff passes a callback function `progressCallback` from `EdenServer` to make `OverlayChekcer` print fsck messages to stdout.
Reviewed By: fanzeyi
Differential Revision: D21966060
fbshipit-source-id: 317b2c8954c718b51b5295c9f8d7698eb203906e
Summary: In its current state, FORCE is a restart mode that cannot be combined with other modes (specifically `graceful`). Removing this coupling from `force` will allow us to at first attempt a graceful restart, but if the daemon is not healthy, then do a force restart, all within one call.
Reviewed By: wez
Differential Revision: D21873166
fbshipit-source-id: 079a777ff6e7b219198cbef15335244b9865f5f3
Summary:
The remotenames extension was enabled everywhere in production. It is
unsupported if remotenames is disabled.
Ideally the cpp tests should use the in-repo configuration instead of
maintaining its own hardcoded hgrc. For now I just did the minimal change to
unblock tests.
Reviewed By: fanzeyi
Differential Revision: D21986274
fbshipit-source-id: b632da074baf4026f331381ae8f2f8479a34e1b3
Summary:
This diff fixes `eden rage` command on Windows.
`eden doctor` and `eden redirect` related rage items are disabled as they are not implemented on Windows yet.
Reviewed By: chadaustin
Differential Revision: D21962673
fbshipit-source-id: 14399a89c26eead8fe5114e0a3764644ee86a1ed
Summary:
Some Thrift clients require that values of type `string` must contain
valid UTF-8, and fail to decode otherwise. Ensure that EdenError's
message is always valid UTF-8.
Reviewed By: fanzeyi
Differential Revision: D21890761
fbshipit-source-id: 93b554e0bc6648c685f5fa54612576bd95d95ae5
Summary:
Add some functions for validating and producing valid UTF-8 to be used
in an upcoming diff.
Reviewed By: fanzeyi
Differential Revision: D21890510
fbshipit-source-id: b25144a34f1df91c72e8ed776b1ee7c1d68344c8
Summary:
The documentation for PRJ_NOTIFICATION_FILE_RENAMED points out that the
filenames can be empty strings to represent files moved to and from the
repository itself. By not special casing this we were trying to create and/or
remove an empty file at the root of the repo which would lead to an overlay
corruption.
Reviewed By: genevievehelsel
Differential Revision: D21961579
fbshipit-source-id: fbfd872ca0377cb90224eba5505ff406812d51a7
Summary: This diff fixes `'__isset' is deprecated` problem when building `//eden/fs/service:edenfs`
Reviewed By: simpkins
Differential Revision: D21966922
fbshipit-source-id: c3b12134f3f84db0a97f3975dae5d64a2c98641a
Summary:
The Python 2-and-3 Thrift API is sort of deprecated and does not
handle binary data in `binary` fields. In advance of migrating to the
modern Python 3 API, remane eden.thrift to eden.thrift.legacy.
Reviewed By: fanzeyi
Differential Revision: D21889697
fbshipit-source-id: a745ee8977999acbfb383a4edebe81d8deb1734e
Summary:
The mount path is required in 2 places: at mount time, and when deleting files.
It turns out that the second one doesn't require it as PrjDeleteFile specify
the filename to be: "A null-terminated Unicode string specifying the path,
relative to the virtualization root, to the file or directory to be deleted."
Thus, we only need it during mount time, so let's pass it to the start method.
Reviewed By: fanzeyi
Differential Revision: D21797080
fbshipit-source-id: 27f5d18841ac077c81ad3f6b3686e20917505c41
Summary:
This will simplify some later diffs where the channel construction doesn't
change any state.
Reviewed By: fanzeyi
Differential Revision: D21797077
fbshipit-source-id: 67d9ee693b067a08986e9bd33f9d91f9a1dadac1
Summary: This is already created, no need for re-recreating it.
Reviewed By: fanzeyi
Differential Revision: D21676314
fbshipit-source-id: 8e3eecfb6069f9bed5bfec7ad6631ed8830d796d
Summary:
These are pure implementation details of ProjectedFs, and don't need to be in
the class, let's move them onto an anonymous namespace in the cpp file.
Reviewed By: chadaustin
Differential Revision: D21676315
fbshipit-source-id: 4fa7088a10256581d7eed2c7cbd90c30a7fef3d2
Summary:
I find this approach to be slighly less error prone due to not having
to manually specify the array size in advance. We lose the field names,
but they should be pretty self explanatory.
Reviewed By: chadaustin
Differential Revision: D21676319
fbshipit-source-id: db4dc993d3dd2a502ae188b08b8d4230939c0415
Summary:
This moves the edenToWinPath inside Windows specific files and makes
the interface clear about the kind of path that is expected.
Reviewed By: fanzeyi
Differential Revision: D21676318
fbshipit-source-id: d153bb543da5f481e94f83d52e45c6cc871d1f54
Summary:
The only reason tests need to set an FsChannel is for calling
cleanupPrjfsCache. For consistency sake, let's do the same as what we do on
unices: verify if we have a valid channel first.
Reviewed By: wez
Differential Revision: D21676316
fbshipit-source-id: 022ca91d32a7ab047a79acccd7d17dfd7906d527
Summary:
The only missing piece on Windows is the bind mount/redirection due to
folly::Subprocess not being present on Windows.
Reviewed By: wez
Differential Revision: D21676313
fbshipit-source-id: a5ba09be04c94b66edf9d40884753afa3865def2
Summary:
We want to use these on Windows too, so let's rename them to not be FUSE
specific.
Reviewed By: wez
Differential Revision: D21676317
fbshipit-source-id: 9f1bde9e0467d8828d4168a69581c8fea54669cd
Summary:
This moves it out of EdenServer and into EdenMount directly. This is closer
to what is happening in the Linux code and thus better.
Reviewed By: fanzeyi
Differential Revision: D21663501
fbshipit-source-id: 84420fcb37586cc2695568f0167b39f91c9bfb63
Summary:
Instead of having 2 different functions for Windows and other unices, let's
only have one. This allows more of the mount code to be shared between both.
Reviewed By: wez
Differential Revision: D21517477
fbshipit-source-id: 185ce864aecdf9a7880a9c705df444ec1e39319a
Summary: This diff updated `eden du` to display a summary and display warnings and cleaning information with colors.
Reviewed By: kmancini
Differential Revision: D21885051
fbshipit-source-id: be127b81c92bea1051a80715682cdbccf22f22e3
Summary: This diff updated `eden du` to display an aggregated result of all mounts instead of showing all details for each mount, as users generally just want to reduce disk usage but don't really care about details.
Reviewed By: genevievehelsel
Differential Revision: D21877178
fbshipit-source-id: dde43e51e96a5c2569c9fe21ab06cc7ea4295866
Summary:
NOTE: This stack works together, they are separated for easier reviewing. Check D21723465 if you want to read this stack in one place.
This diff finally connects EdenAPI with EdenFS. Previously we only had basic EdenAPI implemented but the performance was not acceptable. This is largely due to overheads in talking with the remote server. Now EdenFS is able to import multiple files at once, we efficiently use EdenAPI to import these data now.
Reviewed By: chadaustin
Differential Revision: D21576569
fbshipit-source-id: a45e832ec63d057730138551393ff7547fa2c22f
Summary:
I am planning to start migrating Eden's CLI to the new Python 3 Thrift
implementation. In preparation, slightly clean up the interface and
implementation of our Python 2 Thrift wrapper.
Reviewed By: genevievehelsel
Differential Revision: D21854539
fbshipit-source-id: d398dd3f324c12288871cf0c9db41e64ed4cf7ed
Summary:
The CreateProcess API allows inheritable handles to be available in the spawned
process by passing a flag to it. What the documentation for this API leave to
the imagination is what happens when processes are spawned concurrently in
multiple threads and handles are opened and made inheritable while doing this.
The answer is obvious but the consequences aren't: they are inherited.
When anonymous Pipes are used, one end needs to be inheritable for the spawned
process to being able use it, but if one is created concurrently with spawning
a process, that other process may have an open handle to that unrelated pipe.
And since to detect that a pipe is closed, all handles to the other end needs
to be closed, this can lead to never being able to detect that it is closed...
This scenario is exactly what is happening in eden when spawning the Mercurial
process, and when one of these processes would die, eden would just hang trying
to write to the pipe, not knowing that the process was already gone. To unblock
eden, all the hg.real processes had to be killed, as this would close all the
pipe handles, and then eden would detect that the pipe was closed and re-spawn
Mercurial (only for the exact same thing to happen).
In order to fix this, we need to tell CreateProcess to only inherit the pipes
handles, and nothing else.
Reviewed By: wez
Differential Revision: D21895537
fbshipit-source-id: 3c84a1d0316b50b5750f554fa20f72f59a718882
Summary:
The old `py` Thrift language support doesn't correctly handle string
vs. bytes, which causes an exception to be thrown when deserializing
paths or blobs that aren't UTF-8.
We will eventually want to migrate to the py3 language implementation,
which supports streaming.
Reviewed By: genevievehelsel
Differential Revision: D21693082
fbshipit-source-id: 0ea10fd3960f5acba353bccb83b5cf539e7eeffb
Summary:
In order to start EdenFS automatically at boot, a template service was used
previously, but due to several issues, we decided to move away from it.
Thankfully microsoft supports several other ways of starting tasks at startup,
one of which is the "Task Scheduler" itself.
One of the weird part of the task scheduler is that there isn't a good way
to tell it to not show a console for a non-graphical application, and thus
plainly executing edenfsctl start in it would create a cmd window, which
would then disappear a couple of seconds later. To avoid this, a "graphical"
version of Python is used (pythonw.exe) to start edenfsctl.
Reviewed By: fanzeyi
Differential Revision: D21732281
fbshipit-source-id: 87ef3a2d5569302392bd30a4b9e7fc48807ee316
Summary:
When I was debugging the multithreaded bug issue it came up that the issue would
only affect certain versions of macos. There may be other bugs that come up
affecting only certain os versions. Having the os version in eden rage could be
helpful to identify such issues.
Reviewed By: chadaustin
Differential Revision: D21776154
fbshipit-source-id: a493e7da1823075ca4a845bd73b21716ce884911
Summary: The thrift team is looking to retire `stopWorkers()` and `stopListening()`, so this refactors our code to stop using both of those methods.
Differential Revision: D21851442
fbshipit-source-id: 6d63d435cbd4a51c855bf71a3b2165e35431a4d7
Summary: This diff made `eden du` able to run under any directory and gives statistics for every EdenFS mount on disk by default.
Reviewed By: fanzeyi
Differential Revision: D21846894
fbshipit-source-id: 2d421db8a4a0202419aa7a41e620d92f6ebdea2e
Summary:
Somehow, on Windows, the socket file that was removed during `prepare` may be
back, but not bound, preventing Thrift from binding to it, let's remove it
again since it's not supposed to be there.
Note that this happens 100% of the time when starting EdenFS via the Task
Scheduler at log on, I couldn't reproduce when triggering the task manually,
or when starting edenfs by hand.
Reviewed By: simpkins
Differential Revision: D21755498
fbshipit-source-id: 32a343d9de531b90417dac0aa382a07c85cdee5a
Summary:
The CREATE_NO_WINDOW is documented as:
The process is a console application that is being run without a console
window. Therefore, the console handle for the application is not set.
The last part is what matters here, by default stdin, stdout and stderr are
inherited from the parent process, with this flag, they won't as these 3 will
simply not be set. Removing it allows Mercurial's output to be sent directly to
the edenfs log which will greatly improve our debuggability.
Reviewed By: akrieger
Differential Revision: D21820195
fbshipit-source-id: 63496a1ad28ecf5440c0846d80e1fbda5756b971
Summary: When calling `eden start` from the CLI layer, make sure to redirect stdin in order to daemonize.
Reviewed By: chadaustin
Differential Revision: D21675707
fbshipit-source-id: 26085cc2ff7774e86f03872030c8885bc3c3b949
Summary:
Currently when you run commands on eden on mac that both fetch data for commit
and update to that commit there is a multithreading bug. This could be a new bug
or an old bug that was not causing problems before, but is showing up a lot more
recently.
See http://sealiesoftware.com/blog/archive/2017/6/5/Objective-C_and_fork_in_macOS_1013.html
for a nice explanation of this.
The root cause of this is still unknown, but this flag should bring back the
old behavior where we were not seeing this bug.
Reviewed By: fanzeyi
Differential Revision: D21744987
fbshipit-source-id: 85092e32212e83b16bc00dc0188f03b643c48eea
Summary:
Update EdenFS on Windows to use EdenMain.cpp, the same as on POSIX.
This reduces the amount of code divergence, and also brings several
improvements to Windows: exposing version information over fb303, support for
dynamically changing logging configuration over fb303, etc.
Reviewed By: simpkins
Differential Revision: D21332569
fbshipit-source-id: dd2da7c385e96f65fc3927511c9f84b96bec9e2b
Summary:
Wanting to have EdenFS run as a service on Windows cause a couple of issues
on Windows:
- Needing to log out after installing
- Forcing Sandcastle to use --foreground, which means the edenfs.log is
empty, making it impossible to debug anything on Sandcastle,
- Services can only be started by an elevated user, while the rest of EdenFS
doesn't have this requirement,
- Uses undocumented template services
The next diffs in the stack will attempt to solve all of these by starting
EdenFS as a scheduled task instead, which should solve all the above.
Reviewed By: fanzeyi
Differential Revision: D21732280
fbshipit-source-id: b959344da1e77819220d26695ff6634f13ac4e0d
Summary:
On Windows, if edenfsctl is started by pythonw, sys.stdout will be None,
check if it is before calling isatty on it.
Reviewed By: fanzeyi
Differential Revision: D21732282
fbshipit-source-id: f0a63094f2c53b026c388f4e5b58433be4397ba9
Summary:
The FormatMessage API isn't very well documented in adding \r\n at the end of
the generated message, and the bit of code that needs to be used to remove
these is not very clear either.
We won't get gratuitous empty lines in the log with this.
Reviewed By: chadaustin
Differential Revision: D21663502
fbshipit-source-id: 203ff5015da6f3cebf7eaee9a8f989db342d096a
Summary:
Thrift setter API is deprecated since it doesn't bring any value over direct assignment. Removing it can reduce build-time and make our codebase more consistent.
If result of `s.set_foo(bar)` is unused, this diff replaces
s.set_foo(bar);
with
s.foo_ref() = bar;
Otherwise, it replaces
s.set_foo(bar)
with
s.foo_ref().emplace(bar)
Reviewed By: chadaustin
Differential Revision: D21712029
fbshipit-source-id: 3a332b4bf6fac6b3cf396d34e6d5ca4849181a6d
Summary:
1. Add directory existence checking logic before checking "legacy_bind_mounts_dir"
2. Still print the disk consumption returned by the du command when it fails with an error invoking du -skxc
Reviewed By: chadaustin
Differential Revision: D21683421
fbshipit-source-id: 531b6b289e9ffa445ed691611c9cbf22f393e742
Summary:
This diff is auto-generated to upgrade the Pyre version and suppress errors in eden. The upgrade will affect Pyre local configurations in the following directories:
```
eden
```
Differential Revision: D21687853
fbshipit-source-id: baf0d9bc33f86da63ea289690faca6cf4d566588
Summary: This function won't go ask remote server and is able to produce a result very quick. This avoids the waiting time in the queue.
Reviewed By: chadaustin
Differential Revision: D21586511
fbshipit-source-id: db3793cc4e4e493d1beb6837c0bd91358e7e845f
Summary: Reading from hgcache directly has been rolled out for quite a while now. Let's remove the configuration and make it default behavior.
Reviewed By: chadaustin
Differential Revision: D21585482
fbshipit-source-id: 7f30e262642adf58388a0580aa5a63595fa89155
Summary:
gcc and clang disagree here, so explicitly construct a Future from the
unique_ptr.
Reviewed By: genevievehelsel
Differential Revision: D21687654
fbshipit-source-id: 338174f5d4d051f509faace6e48d1c0fb4d898e1
Summary: I've updated most messages to suggest a graceful restart, but I overlooked this instance and we got a user question about this. So lets update this message here as well.
Reviewed By: chadaustin
Differential Revision: D21681035
fbshipit-source-id: 32bf5e75b9ad1d98bcd70427101aabeee0be5046
Summary: Fix some RVO warnings and a compilation error when compiling with gcc.
Reviewed By: xavierd
Differential Revision: D21481738
fbshipit-source-id: 0621f5886df40c24ef1a6a68ccd957e38f2f4122
Summary:
This script deals with obtaining the right privilege to start the EdenFS
service. This effectively enables `edenfsctl start`.
Reviewed By: wez
Differential Revision: D21585739
fbshipit-source-id: 7b835434d865fa4c4c8473e13665ae669fd86108
Summary: Now that Rust libraries can be linked into fbcode binaries with mode/mac, enable Rust datapack on all platforms.
Reviewed By: xavierd
Differential Revision: D21241075
fbshipit-source-id: ae2abee25a5ad7b9db9ac5e6b8687c5f79376926
Summary: EdenFS doesnt daemonize correctly due to the privhelper not closing fd 0 (see http://www.faqs.org/faqs/unix-faq/programmer/faq/). This redirects stdin to /dev/null/ in order to do so.
Reviewed By: xavierd
Differential Revision: D21602545
fbshipit-source-id: 0aeb589efbf214ef22c0db039fbb6a436a71e360
Summary:
This checks in some design documents based on the "EdenFS Internals"
presentation that I've given internally a few times.
Reviewed By: wez
Differential Revision: D21519530
fbshipit-source-id: 3f79d38e8ccf994b2ef303d491809a91fa5b6d95
Summary:
Next step in unifying the mount path, let's make the initialize the same in
Windows and unices. The only difference is now limited to the .eden directory
which we will be able to implement once regular users can create symlinks.
For the takeover code, the #ifdef is pushed down to the actual code that does
it, this allows the rest of the code to not have to bother about Windows vs
other platforms.
Reviewed By: wez
Differential Revision: D21517478
fbshipit-source-id: d40ca2694d23031ff98e319071e610efa306008f
Summary:
In order to build a StringPiece from a C-string, the length of the that
string needs to be known, and a constexp strlen is performed on it. That
strlen is however a recursive one, causing the stack to blow up on big file.
Interestingly enough, this also means that binary files potentially had a
wrong sha1 being computed, potentially causing `hg status` to report some
files as being modified, while they aren't. By switching to using a vector
instead of a string, the intent should of this should be more obvious.
Reviewed By: simpkins
Differential Revision: D21551331
fbshipit-source-id: 2dc1f08d96f49d310593e0e934a03215be2b5cbb
Summary: Nothing should be running with this version, so this check is not needed anymore!
Reviewed By: chadaustin, wez
Differential Revision: D21513269
fbshipit-source-id: 698c9d84e87461cd0484ef997f44dc9169e87a25
Summary:
While not all stats are supported on Windows, most of them can, let's enable
them. This is a first step in unifying the mount path for Windows and Unices.
Reviewed By: chadaustin
Differential Revision: D21514732
fbshipit-source-id: 00fd89b7499f8ae18c089f98bc18cd43f68538fa
Summary:
While this isn't the right fix, this is what shipped in our packages, for the
sake of being able to reproduce the package, let's land this as it is. A
future change will remove this ifdef.
Below is pkaush original description:
In Eden Windows we treat all the files as regular files and don't have a
concept of symlinks and executable files. Fixing the TreeEntryType::getType()
to return REGULAR_FILE for executable file and symlink.
Reviewed By: wez
Differential Revision: D20481051
fbshipit-source-id: 0b0c4d7aea28134383ef45aeafc02930b420286b
Summary: add two methods calls as typehints (no real code change).
Reviewed By: zeroxoneb
Differential Revision: D21337646
fbshipit-source-id: 8079883f7f989251965d3308c5374f097023b57a
Summary:
- Added uptime field to DaemonInfo thrift struct
- Created startTime member variable in EdenServer
- Made appropriate refactoring changes to EdenMain and EdenServer
- Changed main.py and util.py to use the new uptime value
Reviewed By: genevievehelsel
Differential Revision: D21471140
fbshipit-source-id: 8868de667dfb95de93e3e71b90c0412fb3825388
Summary:
The disabled tests are due to the st_mtime stored in the struct stat only
having a second granularity (as expected), but the tests are changing these
files faster than that, causing them to fail. I'll attempt to fix the file
change detection mechanism on a later diff.
Reviewed By: wez
Differential Revision: D21319025
fbshipit-source-id: a3f2d62a3ce56fea37a62bfad73306ac1556f772
Summary:
Same as the previous one, paths are different on Windows, and thus we need to
test against a different path.
Reviewed By: wez
Differential Revision: D21319021
fbshipit-source-id: 849c86574e69c3f638ea180c3b594aaeae2970ad
Summary:
The use of realpath is needed to resolve a unix-style path to a Windows style
one. This helps keep the test generic with no `#ifdef _WIN32`
Reviewed By: wez
Differential Revision: D21319019
fbshipit-source-id: b8cdd81f0afdd135849a5b850d854399cef8cef8
Summary:
In glibc, pthread cancellation support adds two atomic CAS operations
to each "cancellation point" syscall (see pthreads(7)). This includes
read() and write(). We can avoid that overhead by disabling pthread
cancellation at the start of the FUSE worker threads.
This saves two CAS operations (~40 ns) in the critical FUSE request
processing loop.
Reviewed By: simpkins
Differential Revision: D21469690
fbshipit-source-id: 7f28a2a8e831006351657981e901dc572c58cf48
Summary:
Migrating buck integration to use configurations, which is more performant and is the primary supported way to run Pyre.
Mypy is no longer supported at Facebook, so mypy buck integration is covered in this migration as well.
Configuration target automatically expanded to include all subtargets, expanding type coverage while introducing no more than 2 fixmes per file.
Reviewed By: dkgi
Differential Revision: D21474723
fbshipit-source-id: 2501af8250b4998fe331350b80c9dc8b4ba2dd1a
Summary:
Sometimes, in shell one-liners, it's convenient to insert the current
eden pid. We print the pid as part of `eden status` (like nginx and
many other daemonizing programs), but that is not useful for shell
substitution. So a pid subcommand that succeeds and prints the pid to
stdout if running and fails if edenfs is not running.
An example use would be `strace -fyp $(eden pid)`
Reviewed By: genevievehelsel
Differential Revision: D21397118
fbshipit-source-id: 06010150557ec882b452180665b84452fa06ebbd
Summary:
The information available in `eden top` is growing and as we add more and
more to eden top, we want to make sure that eden top users can understand
what `eden top` is telling them. Particularly if we want eden end users to be
able to debug themselves, providing them enough context on the
metrics eden top provides is important.
The help page explains what data is included, why this data is helpful and
when this data indicates a concern.
Reviewed By: wez
Differential Revision: D20974262
fbshipit-source-id: caaef70e76d283968bee8533c7c34d63a95c5490
Summary:
this is preperation for adding a help page to eden top (D20974262)
This allows adding a scrollable section to `eden top`. This is particuarly needed
so that the help page is readable when it does not fit all in one screen. This
also allows scrolling the process table on the main page to see all the
processes listed (this is added here).
Reviewed By: chadaustin
Differential Revision: D20974245
fbshipit-source-id: 402409efdc451db32e7a267b0750204f2322dc4f
Summary:
this is preparation for adding a help page to eden top (D20974262)
This moves all code for writing to the screen to its own class so that
it is not scattered through out the rest of the eden top logic. This is
primarlity so that it is easier to make changes the screen writing logic to
allow scrolling, which is needed so that the full help screen can be read
when it takes up more space than is avaiable.
Reviewed By: chadaustin
Differential Revision: D20965954
fbshipit-source-id: c2690a293579706605d26be9d3bbe491104e6847
Summary:
Our use of the term "client" to refer to a checkout is
deprecated. Rename some internal functions that use the term client.
Reviewed By: simpkins
Differential Revision: D21395159
fbshipit-source-id: fa96ba593f53b493e5ae816fa686f333a132c232
Summary:
Add new fields to EdenConfig to control log rotation settings, and update
edenfs_monitor to set up the log rotation strategy using these settings.
Reviewed By: chadaustin
Differential Revision: D20427271
fbshipit-source-id: 9960bdb6f4d077a4e21fb4a6209aa02ab21ad653
Summary:
Update the LogFile class in edenfs_monitor to support log rotation when the
file size exceeds a set limit.
There is a generic API to abstract out the log rotation details. For now
I have only implemented a relatively straightforward implementation that
appends the current timestamp to the log path, and prunes old log files.
Reviewed By: chadaustin
Differential Revision: D20427310
fbshipit-source-id: f6aefc8f2fe604fdd1d6a72c3d30666612912f41
Summary:
See the comment in FuseChannel.cpp, but it's currently not easy to
avoid the "security.capability" getxattr request for every
write. Since we can't avoid the request, the fastest thing we can do
is branch, strcmp, and fast-path a result on the same thread.
This appears to save three or four microseconds in the 4k random write
benchmark.
Reviewed By: wez
Differential Revision: D21341973
fbshipit-source-id: a23620767f4bdec4daf02ecfe3acb924dd57857a
Summary:
On Windows, we've had a couple of cases where the overlay is corrupted which
prevent remounting EdenFS. In this case, one entry of a directory was an empty
path. Let's try to catch the error when it happens, instead of after the fact.
Reviewed By: wez
Differential Revision: D21443002
fbshipit-source-id: 47e59aee803cc0b4befd6e0585836d5c68e905b1
Summary:
The disabled tests are due to the precision of st_mtime on Windows only being
a second, but the tests are changing the config faster than that.
Reviewed By: wez
Differential Revision: D21319023
fbshipit-source-id: bcdce24f70ce99984cabb290338ac94a2459e9de
Summary:
These 3 tests compile without issues on Windows. The RocksDB one is weird,
while it compiles with no hickups, I simply cannot run the resulting test
binary, and I'm not sure how to debug this. The local store one fails in folly.
Reviewed By: chadaustin
Differential Revision: D21393724
fbshipit-source-id: db90bf20a9d116bc8aa493703997c5e8da76eb1f
Summary: All the tests are passing.
Reviewed By: chadaustin
Differential Revision: D21341730
fbshipit-source-id: 90a3872b190879ec163935ff53703157028f87bc
Summary:
The modeFromEntryType and treeEntryTypeFromMode tests for symlinks and
executable had to be disabled as these function explicitely do not support
these. Since mode bits are a bit meaningless on Windows, this is probably OK.
Reviewed By: chadaustin
Differential Revision: D21341728
fbshipit-source-id: 86acf24d9ab67a02ecab33b7ebe82a456295fc3c
Summary: All of these tests are passing on Windows with no changes.
Reviewed By: chadaustin
Differential Revision: D21341729
fbshipit-source-id: 2b4d52751e74fa953bfe5143dc0c5735de2d34cf
Summary:
All of these were simply NOT_IMPLEMENTED on Windows, but the code compiles
and doesn't break any existing tests. The underlying called functions might
have been implemented already, or are NOT_IMPLEMENTED, either way, this reduces
the amount of `#ifdef _WIN32`.
Reviewed By: chadaustin
Differential Revision: D21405622
fbshipit-source-id: bdc2de41d6a57e1c0b532e76eeb2c0c86180d558
Summary:
vs2017 is not able to compile the static assertion in KeySpace.cpp.
Previously we thought that this would be resolved in a later release of vs2017
but now that is here it is clear that it hasn't been fixed.
This commit pushes the version requirement to vs2019 (see
https://dev.to/yumetodo/list-of-mscver-and-mscfullver-8nd for a mapping between
product versions and compiler versions), but we cannot build with vs2019
because folly and rangev3 don't compile with vs2019, so this assertion (heh!)
has literally not been tested.
This commit also fixes up an oversight in the gating logic: the intent is that
we perform the assertion on all systems except known broken MSVC. We were
accidentally restricting it to later versions of MSVC.
Reviewed By: simpkins
Differential Revision: D21432890
fbshipit-source-id: e11ffccc53bf8dffdf2db45ad4f3cf199b1cc70d
Summary: Instead of having `HgBackingStore` fetch blobs separately, we now try to read from hgcache and fetch from `HgImporter` as batches.
Reviewed By: chadaustin
Differential Revision: D20903245
fbshipit-source-id: d8e404d6765f1bcbacbf2a39f83eab0a351a3fe0
Summary: This diff makes `HgImportRequestQueue` to be able to return multiple requests in the queue at once.
Reviewed By: chadaustin
Differential Revision: D20197070
fbshipit-source-id: 8cff1780d6e56321a756d30ac0e9b9d5d319c049
Summary: The life of a request is only able to be finished with one of the method. So we can instead having the tracker destroyed when the future is resolved.
Reviewed By: chadaustin
Differential Revision: D20995819
fbshipit-source-id: 5dac2f762513b5d0bcacaab7d0669fc8fdb61e80
Summary:
Readability improvements.
`HgQueuedBackingStore` will need to call these functions individually.
Reviewed By: chadaustin
Differential Revision: D20683321
fbshipit-source-id: 9a9bd766c34559048bd0971f17304090abbb2774
Summary:
During a graceful restart, while the fuse mount is not handling
requests, avoid a possible deadlock between the edenfs, hg
debugedenimporthelper, and watchman processes.
See the comment in HgImporter.cpp.
Reviewed By: fanzeyi
Differential Revision: D21342275
fbshipit-source-id: df8fb5df5d5cd1490e88b42054b34cbb2acdb692
Summary:
On Windows, the following pseudo code:
int fd = open("file");
struct stat st;
fstat(fd, &st);
Will have a different st_dev than the following code:
struct stat st;
stat("file", &st);
Since the FileChangeMonitor uses st_dev as a way to compare if a file
changed, the config is always reloaded.
For our use case, the filesize and its mtime should be enough to know if the
configuration changed, so let's only use these 2 on Windows.
Reviewed By: wez
Differential Revision: D21312679
fbshipit-source-id: f08b3eb7d6037f5d88ece82efe3a5437b1954ba2
Summary:
Only the mtime test had to be disabled due to getMetadata not being tracked
on Windows.
Reviewed By: wez
Differential Revision: D21312680
fbshipit-source-id: 5678956ed2b8b45b38d44c459557161aa6fd222e
Summary:
Fairly straightforward, one change to the test was necessary as reading an
unlinked file cannot be done on Windows.
Reviewed By: wez
Differential Revision: D21312681
fbshipit-source-id: c8cca5eeca7825983176ef618007c514b8e02140
Summary:
This brings it closer to folly::writeFile which should help in avoiding ifdef
whenever we want to use it.
Reviewed By: wez
Differential Revision: D21319020
fbshipit-source-id: 80fbf7fba671b18b5ef68375910e1a2a8869f590
Summary:
On Windows calling `connect()` on an AF_UNIX socket path that does not exist
creates a file at that location. This is problematic as it now prevents
servers from binding to that path. Even if the server attempts to remove the
file in order to bind, clients attempting to call `connect()` can race with it
and make binding fail.
This updates our client connection code check to see if the file exists before
attempting to call `connect()`. This can still race with a server that is
trying to remove an old socket and re-bind, but it at least makes the race
less likely to happen.
Reviewed By: genevievehelsel
Differential Revision: D21410571
fbshipit-source-id: 3df63b19b40b25be98108246186a48a379cdab28
Summary:
The name of a checkout is not a public concept, so refer to the
checkout's path.
Reviewed By: genevievehelsel
Differential Revision: D21393208
fbshipit-source-id: 4c014a6f65515f4632f2dffe5d563d0ee859dda0
Summary:
When I want FUSE stats, I never run `eden stats io`. To unify the
command language, rename `io` to `fuse` and `latency` to
`fuse-latency`, to match `thrift` and `thrift-latency`.
Leave aliases in place in the off chance someone cares.
Reviewed By: genevievehelsel
Differential Revision: D21392931
fbshipit-source-id: 843c1c85ea0aa162ba167f251f0f2cde5a389e72
Summary:
`eden stats memory` was never useful because it shows system
statistics which you can get better from `top`, `htop`, or `atop`.
Reviewed By: genevievehelsel
Differential Revision: D21392737
fbshipit-source-id: 010021b8a97bd8ba8ac289d906acc3c3ecd10768
Summary: So the linter will stop nagging at my changes to this file.
Reviewed By: wez
Differential Revision: D21396011
fbshipit-source-id: dbc3479637cca83aa1a11ff94a9033dfa42fc2a6
Summary:
Talked with xavierd last week and we can use LocalStore's `get_missing` to determine if a blob is present locally. In this way we can prevent the backingstore crate from accidentally asking EdenAPI for a blob, so better control at EdenFS level.
With this change, we can use this function at the time where a blob import request is created with confidence that this should be short cheap call.
This diff should not change any behavior or performance.
Reviewed By: xavierd
Differential Revision: D21391959
fbshipit-source-id: fd31687da1e048262cb4eae2974cab6d8915a76d
Summary:
If config.toml exists but does not contain a valid storage engine,
edenfs would fail to start. Now, it will set a default value and write
the updated config file.
Reviewed By: simpkins
Differential Revision: D19671453
fbshipit-source-id: 32f19dbed2be02bec2a96e1349dca6e7038b9301
Summary:
Addressing issues simpkins brought up on D21207287 when we upgraded and introduced some pyre bugs.
Temporarily upgrading just this project, once we resolve some sandcastle capacity issues we'll release this via another global upgrade in fbcode.
Reviewed By: simpkins
Differential Revision: D21316793
fbshipit-source-id: f0c79f53d97f7182e7d8fe6e081c58ef53ce0c9a
Summary: These docs are all EdenFS specific so move them into the fs/ directory.
Reviewed By: genevievehelsel
Differential Revision: D21329620
fbshipit-source-id: 4090ed4ca371d01ea98e06ad6ce8f434c0660962
Summary:
The MSVC compiler complains that it doesn't have the full definition of
PrivHelper, causing the build to fail. Include the right header to fix this.
Reviewed By: genevievehelsel
Differential Revision: D21381946
fbshipit-source-id: 0d0389ee8db44a36786973404c38487a94e8c4df
Summary: These third-party includes are edenfs-specific, so move them into eden/fs/
Reviewed By: simpkins
Differential Revision: D21314642
fbshipit-source-id: c52b0a00d5080934e1f07e4cd55373602f2f6b0a
Summary: These benchmarks are edenfs-specific, so move them into /eden/fs/
Reviewed By: genevievehelsel
Differential Revision: D21314464
fbshipit-source-id: 1dcf6adfbdea1394f222de4d462397ea531ced00
Summary:
D21316793 is blocked from landing because there are a few targets in eden that are running pyre via buck targets integration.
We can't do custom version overrides for projects that are using a mix of local configurations and buck integration, because buck doesn't provide an interface for setting the equivalent pyre version override.
We're moving away from buck targets integration for pyre across the board, and I've run a codemod over the project to clean up all of the buck typing integration (including some residual mypy) as well as updated type ignores / fixmes accordingly.
Let me know if you have any concerns; upon skimming it looks like most changes are either converting `type: ignore`s into fixmes, or removing `type: ignores`.
Reviewed By: dkgi
Differential Revision: D21343093
fbshipit-source-id: 5ee1436377eb526c0a679fb821c42e07cbca52a5
Summary:
When running Eden as a service, CreateProcess will pop up a console window for every new console based process. Eden fs spawns hg import helper process which will create console windows. It doesn't impact the functionality but is a bad user experience.
Passing the CREATE_NO_WINDOW flag to CreatePeocess will suppress the console Window.
Reviewed By: wez
Differential Revision: D21361412
fbshipit-source-id: 96ecfa2d9ca811287cdc60ba1c4632f16f38340e
Summary: This enables the Edenfs to run as a console application.
Reviewed By: wez
Differential Revision: D21241794
fbshipit-source-id: 704e8488b680ac90f11e9eabef704879a395d7e0
Summary: By default the Eden on Windows will run as a service. It should be installed as a Windows service by the installer to work properly.
Reviewed By: wez
Differential Revision: D21241597
fbshipit-source-id: 2bcbd518d274d829bee5616d266c542f3fcc4b16
Summary:
While EdenFS does not use a separate privhelper process on Windows, it still
defines a stub PrivHelper class. However, this class was previously defined
in a separate win/utils/Stub.h header file, which led to awkward `#ifdef`s to
include the correct platform-specific header file.
This diff moves the definition of the dummy PrivHelper class in Windows into
the same `PrivHelper.h` header file used on POSIX platforms. This results in
a few more `ifdef`s in the PrivHelper files, but fewer `ifdef`s in the calling
code, and will make it easier to start unifying more of the `EdenMain` logic
on Windows and non-Windows platforms.
Reviewed By: xavierd
Differential Revision: D21332568
fbshipit-source-id: c63bf2b4a8b7e767d7db7dcda28675f735c23bf8
Summary:
The kernel has some legacy logic to attempt to clear the setuid and
setgid mode bits upon writes, but it is racy and disabled by default
in libfuse3. Now that we don't support setgid and setuid bits at all,
opt out of the legacy kernel behavior.
Reviewed By: wez
Differential Revision: D21334075
fbshipit-source-id: bdef12c1958a5e9bd2649c2bcb54975b0b4e78d6
Summary:
In the initial stages of the windows port we had
problems building rocksdb on windows, so we disabled it.
These days we're able to build it and detect it--we even
require it in the cmake code, but hadn't gotten around
to telling the rest of the code that we can use it.
This commit re-enables it in the build but leaves sqlite
as the default engine until we're able to perform some
benchmarking.
Rocksdb itself has some build issues on Windows; it doesn't
use cmake to locate dependencies, so even though we built
snappy it doesn't know how to find it without modifying the
source:
https://github.com/facebook/rocksdb/blob/master/thirdparty.inc#L4
For that reason, we disable the use of Snappy in the Windows build.
However, in the version of rocksdb that we were using, it would
default to trying to use Snappy even though it wasn't compiled in
and throw an exception.
I've upgraded to a newer version of rocksdb that will simply not
use compression if no compression was enabled at build time.
Given that we mostly store relatively small objects, I'm assuming
that the lack of compression is fine for now.
Reviewed By: xavierd
Differential Revision: D21319896
fbshipit-source-id: 2a2d06d4bd5382706e9220f9b4a2de99dc18311d
Summary:
Update the Windows code to use the normal StartupLogger.h and .cpp files.
Previously the Windows code defined its own separate StartupLogger.h file.
The class declaration looked largely like a copy of the POSIX
DaemonStartupLogger code, but most of the methods were not actually defined
anywhere. The actual functionality appears identical to the POSIX
ForegroundStartupLogger, so this code just switches to using that in most
cases.
Reviewed By: xavierd
Differential Revision: D21332570
fbshipit-source-id: 0585a38d8f37dc93459d380aa277082c35cbadfc
Summary:
FUSE_HANDLE_KILLPRIV expects that any write() call is handled by
clearing the setuid and setgid bits in the userspace. To avoid
implementing that behavior, disallow setting setuid or setgid in the
first place.
Reviewed By: xavierd
Differential Revision: D21333703
fbshipit-source-id: eb084ee8b00afe74c0da26e41c32c2cb742723da
Summary:
Instead of multiple linear array scans to determine the type of a FUSE
request, use a single lookup table. This is a small optimization, but
it makes the code a bit nicer too.
Reviewed By: wez
Differential Revision: D21314720
fbshipit-source-id: 5b6700ad5bb8e353da1e457de1533e6a626e8f68
Summary:
Google Benchmark is easier to use, has more built-in functionality,
and more accurate default behavior than Folly Benchmark, so switch
EdenFS to use it.;
Reviewed By: simpkins
Differential Revision: D20273672
fbshipit-source-id: c90c49878592620a83d2821ed4bc75c20e599a75
Summary:
Remove the requirement that the `--edenfs` argument flag must be passed in
when invoking the `edenfs` daemon.
This flag was added in D12927803 to help provide a better error message if
users accidentally ran `edenfs` when they really mean to run `edenfsctl`.
However, this shouldn't really be a major problem any more: on Linux and Mac
we now install `edenfs` in a `libexec` subdirectory instead of `bin`, so it
should not be in most user's `$PATH` by default any more. Additionally we
also verify that no other unexpected arguments have been supplied. Prior to
D12927803 the `edenfs` binary would ignore unexpected arguments, making the
error messages more confusing when users tried to run it with `edenfsctl`
subcommand arguments.
One motivation for changing this now is that the Windows version of `edenfs`
does not require this flag, and it is desirable to remove this behavior
discrepancy. Rather than making this flag required on Windows it seems fine
to just drop the requirement on Mac & Linux.
Reviewed By: wez
Differential Revision: D21297159
fbshipit-source-id: e24bd694dadc036cd31dead287ae2c1184747822
Summary:
Move the `UserInfo` code from `fuse/privhelper` to `utils`, and also unify the
POSIX and Windows implementations of this class.
This code was originally put in the `fuse/privhelper` directory since it was
written at the same time as the privhelper. However, it's really a
lower-level library that doesn't depend on FUSE or any of the other code in
the `fuse/` subdirectory.
Reviewed By: wez
Differential Revision: D21296594
fbshipit-source-id: f58682f6ce86bba0328472c491bb4c0dc3370319
Summary:
Enable the unit tests under eden/fs/utils on Windows.
This does comment out a few tests related to `AbsolutePath` that are broken on
Windows. The AbsolutePath constructor does not enforce that the input path is
actually absolute. Additionally the `canonicalPath()` function ends up doing
weird things and returning paths that start with `/` followed by a drive
letter (e.g., `/C:/foo`). These issues should ideally be addressed in
subsequent diffs.
Reviewed By: xavierd
Differential Revision: D21239886
fbshipit-source-id: ef08d62353ba83b96d9fd79bd4636f4a0f961373
Summary:
The operation originally wanted to operate on the fuse `Attr`
structure which we don't have on Windows, so I repurposed the
`InodeBase::getattr` into `InodeBase::stat` and moved the conversion
of `struct stat` to `Dispatcher::Attr` to the `EdenDispatcher::getattr`
method (and a couple of other adhoc places that were doing a similar
conversion).
Reviewed By: chadaustin
Differential Revision: D20562459
fbshipit-source-id: 6b538110038352e9b5590fcb5ff5c33fe84ac1d8
Summary:
The only change I had to make was due to the fact that MSVC wasn't smart
enough to realize that the shift value couldn't be negative, so a manual
folly::to_unsigned was added to silence the warnings.
Reviewed By: simpkins
Differential Revision: D21268634
fbshipit-source-id: e65f15d58d5ea23bfa6796bab23cf1f5c2e7c12c
Summary:
On Windows, the expected refcount is one less than what it is on Linux, due
to the .eden directory not being present.
Reviewed By: simpkins
Differential Revision: D21268203
fbshipit-source-id: 91cfe742fa4d576917d552964d9541dc68ad2c75
Summary: I thought it would be helpful to introduce and `eden uptime` command, especially with automated graceful restart on the way. This prints it in human readable format, later on if for some reason automation would like to use this, a flag could be added that allows for custom formatting. Also, this can be added to `eden rage` output later.
Reviewed By: chadaustin
Differential Revision: D21260800
fbshipit-source-id: 3f9a4f8d6264dfc38bd15c024a0209f7eeb912fa
Summary:
Previously, for logging the value of unexpected enum values, we wrote
`static_cast<int>`. Given enumerations can have any integral type
backing them, this was somewhat inaccurate. Instead, introduce an
explicit enumValue function which returns a value of the appropriate
underlying type.
Reviewed By: genevievehelsel
Differential Revision: D20975176
fbshipit-source-id: 0bb5b0d2f68f8fe9d68e4c6a847d59ae0997d0df
Summary:
D21074489 adds metrics for pending FUSE requests, this cleans up the display for
pending requests.
This removes the max duration of pending requests for FUSE requests since this
data is not available (it is not measured by the FUSE library).
Reviewed By: chadaustin
Differential Revision: D21074746
fbshipit-source-id: e5585ec091aa5fd5499deee2d8be89f47f769a6a
Summary:
This adds the number and duration of FUSE requests to eden top.
The purpose of this metric is to provide a highlevel indicator of
the health of eden and the fuse interface. Because FUSE is the
interface through which the user mostly interacts with the file
system it these metrics are reflective of edens overall
responsiveness.
The metrics are collected per mount, but we only display the
aggregate since this serves as an overall summary metric.
These metrics are displayed similarly to the import metrics,
for consistency. There is not a metric for the max duration
of pending FUSE requests D21074746 cleans this up so that
the NA is removed from the display.
Reviewed By: chadaustin
Differential Revision: D20928603
fbshipit-source-id: ce78f4274203c2302e7595ae2ef601e18893db38
Summary:
This sets up the counters that will allow us to expose the number
of pending FUSE requests in Eden top.
As D20846826 mentions adding metrics for FUSE request gives
visibility into fuse requests and overall health of eden.
This provides more insight beyond the metrics for live FUSE requests
since it shows the kernels view of FUSE requests. Looking at the difference
between the number of pending and live request, can identify issues
that arise at the interface between eden and FUSE and monitor how
quickly fuse workers are processing requests.
**note**: this is only for linux since macos has no equivalent to
`/sys/fs/fuse/connections`
Reviewed By: chadaustin
Differential Revision: D21074489
fbshipit-source-id: c0951f0dfd4fa764be28d8686d08cd0dd807db37
Summary:
It turns out that macOS, FreeBSD, and Linux all agree on the values
for dtype. If we assume dtype is just the high nibble of mode_t
shifted right by 12, then the Windows CRT agrees too. So hardcode the
values in eden.thrift and add appropriate static_asserts. This opens
the possibility of simply static_cast'ing the numeric values in
Watchman.
Reviewed By: simpkins
Differential Revision: D20975101
fbshipit-source-id: 354ffcbdf3f1d5f8b1715abf1026eaea429d16cf
Summary:
This makes it a bit easier to write integration tests
that run an isolated EdenFS instance complete with its own
unique scratch path environment.
Reviewed By: genevievehelsel
Differential Revision: D21258455
fbshipit-source-id: 3ea13b6337bb8bb58a237e67488b116c423bbe6c
Summary:
In the past, if an EdenMount got stuck while shutting down (inodes still referenced), the EdenMount would already be removed from the server's `mounPoint's` map while stuck. This resulted in `eden list` showing that the mount was not mounted, and then if we tried to remount the mount, we'd get a confusing error message about the not being able to grab the overlay lock. This also made it so if we tried to attach with gdb, or if we core dumped and tried to look at the data, we wouldn't be able to examine the object.
Instead, we should only remove the mount from the `mounPoints_` map after it has completely been shutdown. This will allow us to get better signal on the state of the mount (it will be in the `SHUTTING_DOWN` state in `eden list` instead of `(not mounted)`) and it will allow us to better debug these situations (where a mount is stuck shutting down) since we can get more information about the mount that is in this state.
Reviewed By: simpkins
Differential Revision: D21211544
fbshipit-source-id: 97ff11df67d0a80f47cf6b9f563876e17bd76622
Summary:
enums and i16 are not serialization-compatible in Thrift, but we want
an enumeration with defined values for dtypes. Rename the existing
Dtype type to OsDtype in preparation for introducing the Dtype enum.
Reviewed By: simpkins
Differential Revision: D20974559
fbshipit-source-id: 5d9087a79e7fd59b99b257a99351a8287ba00b5f
Summary:
From looking at the code, it appears that the FUSE refcount is always a
uint32_t, except when serialized. Let's do the signedness conversion and
narrowing then.
Reviewed By: simpkins
Differential Revision: D21240161
fbshipit-source-id: 877c6cb6881cb36346c64cf92d99b1e588aed580
Summary:
The disabled test uses chmod and symlinks that aren't present in the WIN32
build. I'm not exactly sure why the .eden couldn't be found though.
Reviewed By: simpkins
Differential Revision: D21264070
fbshipit-source-id: da7385d9de4002c99fcbe79f37bd09acd41fbec8
Summary:
Add a getEntryInformation Thrift call that allows looking up existence
and dtype without fetching blobs. There is still plenty of room for
improving this - we don't need to allocate inodes for every path to
satisfy this query.
Reviewed By: genevievehelsel
Differential Revision: D20974514
fbshipit-source-id: e7fe5b6bd789aaadbb8295c3d69099cb2f17debd
Summary:
We have a lot of warnings on Windows, which in case of a build failure makes
it hard to find the actual cause, let's start by silencing the Sqlite ones.
Reviewed By: wez
Differential Revision: D21238029
fbshipit-source-id: 95763dcd06af77d45b7a769a93e0bbd6cabfa9c3
Summary:
At the end of the test, the test assumes that an old uncommited version of
the file can be retrieved thanks to the files living in the overlay. On
Windows, the files are written directly onto the Projected FS cache, and thus
the old version of the file cannot be retrieved.
Reviewed By: pkaush
Differential Revision: D21193209
fbshipit-source-id: a4db6fb2bfc68817601725f97a6d47f2319b83c7
Summary:
While I had to disable a bunch of them at first, this allow them to compile
properly and run. Future diffs will attempt to enable the disabled ones.
Reviewed By: pkaush
Differential Revision: D21188118
fbshipit-source-id: 154fec49c76563b0856fa36e78b2bbd09f0f2381
Summary:
Now that we have globFiles available via thrift on windows,
we can use it to implement eden prefetch on windows, rather than the
significantly slower `hg prefetch`.
Reviewed By: pkaush
Differential Revision: D20781187
fbshipit-source-id: f6cd37a2a504d07f03be2695a69a72ddf7a62593
Summary:
Add variants of the helpers that return by value rather
than by reference so that we can use those on Windows (which
doesn't embed a `timespec` in a `struct stat`).
Reviewed By: simpkins
Differential Revision: D20562242
fbshipit-source-id: e4769fccb40229765bbf99d0967708cc864db6c3
Summary:
This enables globFiles for Windows, with some
minor tweaks around dtype to enable the build and make
the results consistent between watchman and eden.
Reviewed By: chadaustin
Differential Revision: D20536715
fbshipit-source-id: b1c8184dc664910e4d052a21b4cd993ddfaadf25
Summary:
This is ostensibly removing and ifndef to enable watchman
to successfully query the journal.
However, for this to work we need to teach the mountPoints list about
normalizing file names.
We avoid using realpath on posix systems because we absolutely cannot afford to
make VFS calls back to ourselves. I don't know if the same applies on Windows.
If not then perhaps the right thing to do here is to use something like
GetFinalPathNameW to canonicalize the paths.
For now, this does the dumb thing and lowercases and converts slashes.
With this change, watchman is able to subscribe to eden and query
journal information.
However, since most queries get translated to glob requests it
still isn't useful; need to enable globFiles in a follow up diff.
Reviewed By: pkaush
Differential Revision: D20516737
fbshipit-source-id: 2053d7b565a40f6ae9cc738386e37d737772e91e
Summary: This diff ports DiffTest on Windows and enables them for the CMake builds.
Reviewed By: simpkins
Differential Revision: D20480869
fbshipit-source-id: 77eda5ef53e4cd19552f0da52de4ff53d9807cc3
Summary: This diff adds readdir tests for Windows and enables InodeBaseTest and TreeInodeTest for CMake build.
Reviewed By: simpkins
Differential Revision: D20480866
fbshipit-source-id: 44fb33448c2e6cdb3ecce08136bf8b9e1d5c398a
Summary: This diff make the InodeMapTest work on Windows and enables this test on all the platforms for CMake build.
Reviewed By: wez
Differential Revision: D20480865
fbshipit-source-id: 8f13f88bb3655475c635c889851e39672ce5b9c7
Summary:
Enable building fake_edenfs.cpp and TakeoverTool.cpp in the CMake build.
This includes a few changes to get `fake_edenfs.cpp` closer to building on
Windows, but at the moment it still does not fully build since the
`StartupLogger` is currently not compiled on Windows.
TakeoverTool.cpp is specific to the graceful restart tests, and doesn't make
sense to ever build on Windows.
Reviewed By: wez
Differential Revision: D21214614
fbshipit-source-id: a7a8306ea3f69579fa02bd3753a66d3095ea004a
Summary: This diff enables running source control update and rebase operations on Eden Windows clones.
Reviewed By: wez
Differential Revision: D20480873
fbshipit-source-id: 9e7cd6acdd9aedc87ddfe126718f48509d1d7c4f
Summary:
This diff updates the EdenDispatcher to fetch/update the FS info in EdenMount which is backed by InodeTree. This helps to store the FS state in the Inode structure, which are used for source control status and update operations.
Also added a custom formatter definition for RelativePathPiece to make it easy to log relative paths.
Reviewed By: simpkins
Differential Revision: D20480861
fbshipit-source-id: b4bf1da3eeebeaee46a4a187eea9193302182068
Summary: On Windows we are seeing frequent hang issues while using telemetry hg binary. Switching to using the hg.real for all the platforms.
Reviewed By: simpkins
Differential Revision: D20480875
fbshipit-source-id: 82892da7f8d46f2413442c37251243400180f9db
Summary: Eden Windows now shares the common EdenMount code. We can remove the unused version of EdenMount.
Reviewed By: simpkins
Differential Revision: D20993231
fbshipit-source-id: 943ee540c5f2aa30827afb73658cb0250143e4af
Summary: Now we are building the merged EdenMount on Windows, so we can also start using the new TestMount code.
Reviewed By: simpkins
Differential Revision: D20966556
fbshipit-source-id: a726b13250d6c29872349e1a2f562c719420f3f1
