Summary:
This is a rough pass that resolves a linker issue on MSVC by
switching to inline static member functions.
Reviewed By: chadaustin
Differential Revision: D20529163
fbshipit-source-id: 578ed440758c685091d3e039e261638e027db17a
Summary:
the type of `mode_t` that we were using for this on Windows
recently changed in the folly portability layer, resulting in a padding/alignment
problem that breaks our assumption at the size of this struct.
I don't believe that we were using bitfields correctly here; in my experience
all of the members must have the same type in order to get packed in the
struct correctly, because the standard doesn't define much in the way
of bitfield behavior beyond the syntax.
This commit uses an unsigned integer type for all of the bitfields
and takes a stab at using a cast when returning the actual mode
member for macOS. This may not be right; I'm just throwing up this
diff first in order to be able to get it on a mac machine.
Reviewed By: simpkins
Differential Revision: D20529006
fbshipit-source-id: e9362bf263fab03d51cc8aa97f94f00806650618
Summary: This diff bumps the open call from FUSE to High priority (which is higher than any other blob open request atm). This has shown improvement on the user experience of EdenFS when it's importing many other things from other channels (thrift, etc.)
Reviewed By: chadaustin
Differential Revision: D20287389
fbshipit-source-id: 319bc44ef8be5c904d7cf0db7cc2f8be28b4760a
Summary: We should disallow `checkout` calls to queue. In practice, this is hard to encounter because of the Mercurial repository lock, but edge cases can slip through (user running a long `hg up` and control-cing the command and then running `hg up` again). There is not a use case in which `checkout` calls should queue, and keeping the Thrift queue clear of unnecessary (possibly) expensive commands should generally help user experience.
Reviewed By: simpkins
Differential Revision: D19894583
fbshipit-source-id: 2b6f0d6bd1f801872fa6fa66dc7728dc9c83d158
Summary: Moving these two files to integrate them better with rest of the Inodes code.
Reviewed By: simpkins
Differential Revision: D19956267
fbshipit-source-id: 5bf8165a6dc70a0c03f71f3f73b383991f21eb56
Summary: Windows sdk doesn't define mode_t. On Windows we use mode to store the file type info which should fix in 16 bits - S_FMT is defined as 0xf000.
Reviewed By: simpkins
Differential Revision: D19956270
fbshipit-source-id: 81acd2cd708f1c82674b8b75cd607d4f5b41a156
Summary:
Migration from Future-returning executor-erasing collectX forms to
SemiFuture-returning forms, that are less risky in particular with coroutines.
Earlier diffs added SemiFuture and Unsafe versions. This codemod migrates
collect versions to the Unsafe versions to allow the basic collect versions to
be made safe.
Reviewed By: simpkins
Differential Revision: D20331206
fbshipit-source-id: efc8dff487d45f7d53ee55e8c4696bd3eed0e6da
Summary:
The result of `errnoStr` is often converted to `std::string` so returning `fbstring` adds an extra copy. Make it return `std::string` instead. This will also allow removing dependency between `String.h` and `FBString.h`.
(Note: this ignores all push blocking failures!)
Reviewed By: yfeldblum
Differential Revision: D20195395
fbshipit-source-id: 0dc65f1566911156be3fcb715dd105c58f2a8822
Summary: Pass a valid ObjectFetchContext down into certain untracked requests.
Reviewed By: chadaustin
Differential Revision: D20243575
fbshipit-source-id: e7112c3bab1265803a26130c4d72905c25f2e729
Summary:
Eden SCM expects that DRY_RUN reports the same conflicts as a normal
checkout, but EdenFS would skip traversing deleted trees in dry run
mode. Fix that and add a test.
Reviewed By: genevievehelsel
Differential Revision: D19782543
fbshipit-source-id: 7a269e67a41b7ad6ce6c54fde37e8f74fcc1ef51
Summary:
This commit causes a desktop notification to be shown if we generate
EIO or ETIMEDOUT responses via fuse; the prompt is intended to make it obvious
to the user that they need to connect to the VPN.
The commit by itself doesn't show a notification, it allows configuring a
command that can be run to do something to show a notification.
The test plan includes one such configuration for our corp environment.
* It doesn't trigger for thrift-originated downloads (eg: prefetch), only for
VFS operations through FUSE.
* Ideally we'd know exactly when we have a network related error in the store
code and use that to trigger the notification. However, we have a rather
convoluted set of importers and fallbacks today, one of which is interpreting
a generic response returned from a pipe, so it is not especially clear
exactly where we should locate the logic
Reviewed By: chadaustin
Differential Revision: D17513364
fbshipit-source-id: 45134f3672679cb5580cb0c1bc12a0d6e38525ca
Summary: after reviewing my stack D18647089 , chadaustin recommended some documentation surrounding this area since it is a bit more complex now, and since it was easy to deadlock and there are a lot of moving parts, I added documentation about how the diff path works, including the new code path, and other non-trivial parts.
Reviewed By: chadaustin
Differential Revision: D19688435
fbshipit-source-id: f6512340421e7c25d4fb4d27177c6187eb57ccbf
Summary: In addition to duration and success, log object fetch counts and checkout type to Scuba.
Reviewed By: fanzeyi
Differential Revision: D19334276
fbshipit-source-id: dabf52427f2ebda2b58df93194df39d52f4fcb4f
Summary: Log the number of object lookups and cache hit rates for a checkout operation.
Reviewed By: simpkins
Differential Revision: D19191201
fbshipit-source-id: 5e9ad501e704810f072dabcda3fce86d027c452e
Summary:
During checkout and stats, count every object fetch and which level of
cache it was served from.
Reviewed By: simpkins
Differential Revision: D19186333
fbshipit-source-id: fc0a74db297b9c723682e245996a7befd762f933
Summary:
Add a fetch context interface to ObjectStore that allows tracing cache
hits, backing store fetches, and fetch durations in the context of a
diff or checkout operation.
Reviewed By: simpkins
Differential Revision: D19135625
fbshipit-source-id: d0d8f134b1c89f7ba4971a404a46a69a1704ba5c
Summary:
There were some unsafe uses of folly::collect, especially in
the diff code. Switch to collectSafe.
Reviewed By: simpkins
Differential Revision: D19529784
fbshipit-source-id: e0ac920a2d2b44767a453f2c41a9166e72246ab6
Summary:
When importing an object from the backing store failed, we would not
fail lookup requests (e.g. lstat) so that the file could still be
removed with `rm`. Unfortunately that data would get cached, even if
the error was transient. This diff returns a correct inode number for
failed requests and limits the amount of time these fake stat()
results are cached.
Reviewed By: wez
Differential Revision: D19655425
fbshipit-source-id: 1a44ace9217f1ab3f69a2c5a6edc89d0f9c4a941
Summary:
I have a suspicion that most edenfs starts are unclean. To test that
hypothesis, add a "mount" structured event type that records whether
the mount was cleanly shut down, how long mounting took, and whether
it's a graceful restart.
Reviewed By: genevievehelsel
Differential Revision: D19585161
fbshipit-source-id: 16eedfeb6181742e64b45bca1ed3cce3edb663cb
Summary:
In any case in which our Inode is not materialized and we would like to continue to diff, we can take the hashes of both the Inode and the ScmEntry and start using the source control tree Differ.
Also, this allows us to remove the RemoveDiffEntry since that only looked at the scmEntry, so that work can also be deferred to the source control tree differ easily.
This avoids loading inodes when we don't need to. In the case of a clean checkout with two commits 30000 commits away, we reduced Inodes loaded by 98.6%.
Reviewed By: chadaustin
Differential Revision: D19292528
fbshipit-source-id: 7b5ec161f7bd65962ae30c49fdad6dbc73954c2a
Summary:
In any case in which our Inode is not materialized and we would like to continue to diff, we can take the hashes of both the Inode and the ScmEntry and start using the source control tree Differ.
This reduces the number of inodes loaded by avoiding to load them if we don't need to. In the case of a clean checkout with two commits 30000 commits away, we reduced Inodes loaded by 98.6%.
Reviewed By: chadaustin
Differential Revision: D19292527
fbshipit-source-id: 2dbf6c4130214f581b6cc09e663f7567abab6ef3
Summary:
In any case in which an inode is not materialized and we are computing a diff, we can take the hashes of both the Inode and the ScmEntry and start using the source control tree Differ.
This avoids loading inodes when not necessary. In the case of a clean checkout with two commits 30000 commits away, we reduced Inodes loaded by 98.6%.
Reviewed By: chadaustin
Differential Revision: D18647087
fbshipit-source-id: 74632de777f04e2468d10977cc757b8993d439e9
Summary: Pass a GitIgnoreStack* and isIgnored flag through the diff operation. It is used later in the stack when we go down this code path in the TreeInode diffing code. When context->loadFileContentsFromPath() is null, the gitignore loading code will be ignored, used when we do not want to honor a gitignore file. For example, during `getScmStatusBetweenRevisions()`.
Reviewed By: chadaustin
Differential Revision: D18647089
fbshipit-source-id: 20d2abd2ef61669465e134165da5a0ac5e987cca
Summary:
In the case that the overlay was closed before the initial
createOverlayFile, prevent an invalid OverlayFile from being accessed,
causing the test to fail.
Reviewed By: genevievehelsel
Differential Revision: D19591636
fbshipit-source-id: 9d961f044c3b68b5c0b2dcd108ff85db6326276d
Summary:
stderr: eden/fs/inodes/EdenMount.cpp:1122:37:
error: local variable 'unmountError' will be copied despite being thrown by name [-Werror,-Wreturn-std-move]
eden/fs/inodes/EdenMount.cpp:1122:37: note: call 'std::move' explicitly to avoid copying
Reviewed By: simpkins
Differential Revision: D19408659
fbshipit-source-id: 1575104414daaeb06892586492cdb74e2df604c5
Summary: based on comment on D18647089, changes the behavior when a directory is replaced with a file, and then the directory is ignored. The behavior should be that the file is reported as `ADDED`, previously the file would have been reported as `IGNORED`
Reviewed By: simpkins
Differential Revision: D19318796
fbshipit-source-id: 8da1b7c6df182ca591e4aea21f6573ca81790cb4
Summary: scuba logging for mismatched parent commits, it would be nice to have a count on how often this is encountered so we can pinpoint spikes of this case. I am not sure how helpful having the parent commits would be in the event, my thought there was that it could help see how far away the commits are and if one of the commtis doesn't exist (like in the case of stripping commtis with a cancelled hg split), but I am open to having an empty event as well if storing these strings is too expensive.
Reviewed By: fanzeyi
Differential Revision: D19432378
fbshipit-source-id: fa3e7cd6aef832c4f918f339b781590b7484cfdc
Summary:
Reduce the log level for a few messages to help reduce some less important
clutter when running with an elevated debug log level.
Reviewed By: chadaustin
Differential Revision: D19188773
fbshipit-source-id: 396bb15e119fc12765ecbc707e7a1dadbdd02422
Summary:
Add a config option to restrict the number of tree prefetches that can run in
parallel. Without this applications that recursively walk a checkout tree can
end up spawning a huge number of asynchronous prefetch operations. These
prefetch operations are quite expensive today, as we currently have to fetch
the full file contents in order to retrieve the file sizes. A recursive
directory walk can end up building up a huge backlog of in-progress tree
prefetches. This will slow down the directory walk, and it can take tens of
minutes to finish processing the prefetch backlog even after the directory
walk is aborted.
Reviewed By: chadaustin
Differential Revision: D19136685
fbshipit-source-id: cdc7a570d016fd7ca81a639cef83380b197acdfc
Summary:
EdenMount offered both getRootTree and getRootTreeFuture. Replace the
former with the latter and adjust remaining callers.
Reviewed By: genevievehelsel
Differential Revision: D18666585
fbshipit-source-id: 961361a1fb545e48db0ece970bb5e6ffbca8efef
Summary: was looking in this area and saw this TODO. moves the `setParentCommits()` call into `CheckoutConflict::finish()` so it is called while we are still holding the parents lock.
Reviewed By: chadaustin
Differential Revision: D18668329
fbshipit-source-id: 8415c792193e5b89737f15141f33c5a9799f527b
Summary: Saves loadFileFromPath in the DiffContext. This allows the source control differ ( at `eden/fs/store/Diff.cpp`) to call this function while it is trying to load gitIgnores and resolve symlinks.
Reviewed By: simpkins
Differential Revision: D18647088
fbshipit-source-id: bddff0158130f0c96ac263bcce81dc51fc017800
Summary: Adds a helper function that can take a path to an Inode and load a file by simply getting the Inode then calling the existing `loadFile()` function
Reviewed By: simpkins
Differential Revision: D18647090
fbshipit-source-id: 89c25bef10a7a0d0ffd660293f85479ac0e9f9a9
Summary: Move the bulk of the gitignore loading to EdenMount, this is needed later in the stack to pass this function as a callback through the DiffContext to enable gitignore loading from the source control tree differ.
Reviewed By: simpkins
Differential Revision: D18565265
fbshipit-source-id: 062349d81bbfb78235ccb34d6ec72ad445a6e2b0
Summary:
Each entry in InodeMap::unloadedInodes_ stored the InodeNumber
twice. Remove one of them.
Reviewed By: wez
Differential Revision: D18651007
fbshipit-source-id: be85c34cb2b38fc0b2875d0874cecd1ef274aca4
Summary: Small readability improvements I noticed while reading the diff algorithm.
Reviewed By: genevievehelsel
Differential Revision: D18650987
fbshipit-source-id: 19346362711455ae87d6770812e647544a4576bf
Summary:
Add default case handling for some switch statements that do not handle the
case of an enum variable being set to something outside of one of the defined
values.
Reviewed By: fanzeyi
Differential Revision: D18652102
fbshipit-source-id: abcb3f9dc1f6b0dc761170c7115ddd9ad5145bc4
Summary:
This splits `EDEN_BUG()` into three separate version. All three crash in
debug mode builds, but in release builds they behave differently:
- `EDEN_BUG()` throws an exception
- `EDEN_BUG_FUTURE(Type)` returns a `folly::Future<Type>` that has been
fulfilled with an exception.
- `EDEN_BUG_EXCEPTION()` returns a `folly::exception_wrapper`.
The main advantage of this is that this allows the compiler to detect that
`EDEN_BUG()` can never return. Previously `EDEN_BUG()` was used for all 3 of
these different cases, and its behavior depended on whether `toException()`
was ever called. As a result we could not easily get the compiler to identify
code paths where we know at compile time that it will never return.
Reviewed By: chadaustin
Differential Revision: D18652103
fbshipit-source-id: 070107c7520f51b05696905fa243de5f8df15958
Summary: Even if blobs have different hashes, they could have the same contents. For example, if between the two revisions being compared, if a file was changed and then later reverted. In that case, the contents would be the same but the blobs would have different hashes. Currently, `getScmStatusBetweenRevisions()` would report false positives in this case. This is also needed so we do not report false positives in `getScmStatus()` when hit this code path
Reviewed By: simpkins
Differential Revision: D18647086
fbshipit-source-id: 66e12648a24fd7e5612eee5e599a5b81c7c5f2d1
Summary:
Record the durations of each phase of checkout so it can be logged to
scribe in a future diff.
Reviewed By: simpkins
Differential Revision: D18632258
fbshipit-source-id: e447976af22cf25cffd640e79b95da2989b07448
Summary:
We're about to return more information from checkout, so
introduce a CheckoutResult struct.
Reviewed By: genevievehelsel
Differential Revision: D18630820
fbshipit-source-id: 78b68ef3551ca8c9e9968631de554084c333b4b9
Summary: make the error message returned in the case of out of date parents during a new status call more user friendly and provide possible remediation instructions
Reviewed By: simpkins
Differential Revision: D18328835
fbshipit-source-id: b214f45bb055d008db8b233ddd2a1843332db838
