Summary:
Add a standalone program that can be used to garbage collect the local store
data when edenfs is not currently running.
Reviewed By: wez
Differential Revision: D14910392
fbshipit-source-id: 1cf8f959e5967cfbb797752653cc72223c87134f
Summary:
Let's catch more Xcode and upstream regressions by enabling more
builds and tests on macOS.
Reviewed By: simpkins
Differential Revision: D15003839
fbshipit-source-id: 257086d578f28198c7718906e1ec06395a55896e
Summary:
Build the store tests on mode/mac. RocksDB doesn't build yet (and we
will likely move away from it anyway) so split it into its own target.
Reviewed By: simpkins
Differential Revision: D15002840
fbshipit-source-id: fa0fea1d52df06526032a6fd585e2da7273910ef
Summary:
The hg store implementation depends on Rust code that doesn't build on
mode/mac yet, but TestMount's dependency on hg was false anyway.
Reviewed By: simpkins
Differential Revision: D15001077
fbshipit-source-id: 840fdf565093ad7a6eb95e001a74ae95eecbf1db
Summary:
Some Eden users observed that eden doctor wasn't cleaning up the stale
mounts left from a crashed (or oomkilled) edenfs process.
I reproduced by running eden doctor, killing the privhelper, then the
main edenfs process, then running eden doctor again.
Reviewed By: simpkins
Differential Revision: D14991043
fbshipit-source-id: 72e771bd556424b5c9da6be178afbf337fbfe4ad
Summary: These allow more tests to compile
Reviewed By: chadaustin
Differential Revision: D14994582
fbshipit-source-id: 6b2a0b276fda64c7f27e28ea9e6d548aaaa1db7c
Summary: This diff add four metrics to the stats we are collecting in Eden. Both time cost on calling `get_tree` and `get_blob` from hg and mononoke backing store.
Reviewed By: chadaustin
Differential Revision: D14780349
fbshipit-source-id: f0e902c7f7523fac2ae576a67f0c85555064e26c
Summary:
The Finder shows the volume name instead of the directory
name when it renders the directory containing a mount, and the default
name of `OSXFuse Volume X` makes it hard to distinguish between multiple
mounts in the same folder.
This changes the volume name to be the basename of the mounted path so
that the finder renders things more reasonably.
Reviewed By: chadaustin
Differential Revision: D14969807
fbshipit-source-id: de9d8469360712828598fed124ed02ee5d6dd44a
Summary:
Unfortunately, some node modules packages include `.DS_Store` files
and since we disable apple double, the kext refuses to allow them to be
unpacked in the repo. Ideally they wouldn't be in the repo, but we need to
unblock that workflow.
Reviewed By: chadaustin
Differential Revision: D14969808
fbshipit-source-id: f9262294a223fd5cc7929a2b324f2401e1fed083
Summary:
Replace Future::onError with Future::thenError:
* to remove ambiguous typing
* to ensure that the executor is not lost and the returned Future is still bound to an executor
See:
https://fb.workplace.com/groups/fbcode/permalink/2002251863144976/
for details.
Reviewed By: Orvid
Differential Revision: D14975808
fbshipit-source-id: b7790bf0e8965c9ffb3e63901bcdaf108480d4ca
Summary:
Add a helper function to recursively remove a directory from the local
filesystem, given an `AbsolutePath`.
Reviewed By: wez
Differential Revision: D14954737
fbshipit-source-id: a3f6b60e610c18eabea2a7c507661a43502a2554
Summary:
Add a new class to manage access to the .eden state directory, and move some
logic for managing this director from the `EdenServer` class and into this new
class.
This is largely just a first step. In the future I think we should move
additional functionality into this class. For instance, logic to determine
the correct local store type and logic to load the checkout map from
config.json. In the future it probably also make sense to move this class
from the `service` library into a slightly lower-level library (perhaps
`config` or `store`).
Reviewed By: chadaustin
Differential Revision: D14889617
fbshipit-source-id: 15cb7daa9b97500124152d016bfdd686d3039771
Summary:
Move some of the argument parsing and config setup code out into a new
EdenInit.h header file. This makes it possible to re-use this logic for other
standalone utilities that want to be able to find the Eden state directory and
config information.
For now I have updated the `fake_edenfs` helper tool used by the integration
tests to use this. This may also be useful for writing standalone tools that
can perform garbage collection of the LocalStore or checking of the overlay
state.
Reviewed By: chadaustin
Differential Revision: D14889616
fbshipit-source-id: b0b193a42cb2f52177d0c44592426b42e27242aa
Summary:
This effectively reverts D14452214, which caused Eden to write an `id` entry
to each RocksDB column family and then flush the column family each time
edenfs started.
There was relatively little benefit to this in practice. It only matters in
cases where the RocksDB column families never had enough data written to them
to get flushed automatically and then a repair is required.
On the other hand it does have some material downsides: it flushing the column
families can be fairly expensive, and can require a substantial amount of free
disk space. This flush caused some users to not be able to start up edenfs
when they did not have enough free disk space.
Reviewed By: chadaustin
Differential Revision: D14947235
fbshipit-source-id: a29f98163fa87185b028bb47945b6fab75700fd6
Summary:
When counters are renamed (port, trunk or queues), the current FBOSS logic
creates a new monotonic counter with new name and removes (last reference to)
monotonic counter for the old name.
However, the old counter continues to unncessarily get exported. This patch
fixes it by deleting the old counters.
Differential Revision: D14942193
fbshipit-source-id: 34cf7dd5290ae87a2c8002775e9d0940615c47fb
Summary: Fallout from another diff that never got landed.
Reviewed By: simpkins
Differential Revision: D14938191
fbshipit-source-id: 980728ca6a1d4c4f90fea46aac9525a03d9809e5
Summary:
Change the `eden gc` implementation to use RocksDB's `DeleteFilesInRange()`
function, as well as its new-ish `DeleteRange()` method.
This makes the garbage collection much faster, and also require much less free
disk space than previously.
`DeleteFilesInRange()` asks RocksDB to simply delete all SST files from disk
if they only contain keys in the specified range. Since the range we specify
should include all keys in the DB this should simply drop all SST files for
this column family.
We also call `DeleteRange()` after this, just in case. This API is relatively
recent, and writes a single tombstone saying that the specified range has been
deleted.
Reviewed By: wez
Differential Revision: D14910345
fbshipit-source-id: c76bdc1c8e07cb2def66673ea892e7f455c9dc7a
Summary:
On my system this test was failing because the `systemctl start exit.target`
command was exiting with an error. It looks like this command exits because
systemd closes its D-Bus connection as it is shutting down. This modifies the
test to ignore the return code from this systemctl command.
I also updated the log message when shutting down systemd to no longer include
the exception backtrace. This message is always logged when running this test
(since systemd is not actually running), and the backtrace information simply
makes the test logs harder to read.
Reviewed By: wez
Differential Revision: D14886191
fbshipit-source-id: 87c996b2579a9920a72ee5b57608c263ca080d6e
Summary:
EdenStats is currently an alias for ThreadLocal<EdenThreadStats>. I want to split EdenThreadStats into two structs which are allocated independently, but EdenStats's interface makes this impossible.
Refactor EdenStats from an alias to a class and encapsulate the underlying ThreadLocal<EdenThreadStats> member.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D14822272
fbshipit-source-id: 691f4731aa22ecbdcd3535ee0bb0b99c816ffc3d
Summary:
I'm confused by the naming of EdenThreadStats and ThreadLocalEdenStats. For example, when I see "ThreadLocalEdenStats", I think that such objects can only be accessed by one thread. That's definitely not what "ThreadLocalEdenStats" means!
I think the following naming scheme makes more sense:
* **EdenThreadStats**: Statistics which are updated by one thread. Currently called EdenThreadStats.
* **EdenStats**: Statistics for all threads. Provides access to EdenThreadStats. Currently called ThreadLocalEdenStats.
Implement my preferred scheme: rename ThreadLocalEdenStats to EdenStats.
This diff should not change behavior.
Note: Prior to D14822274, EdenThreadStats was called EdenStats.
Reviewed By: simpkins
Differential Revision: D14822271
fbshipit-source-id: bd20179b1010588e3fc16dc9ed0657d458606f16
Summary:
I'm confused by the naming of EdenStats and ThreadLocalEdenStats. For example, when I see "ThreadLocalEdenStats", I think that such objects can only be accessed by one thread. That's definitely not what "ThreadLocalEdenStats" means!
I think the following naming scheme makes more sense:
* **EdenThreadStats**: Statistics which are updated by one thread. Currently called EdenStats.
* **EdenStats**: Statistics for all threads. Provides access to EdenThreadStats. Currently called ThreadLocalEdenStats.
Implement half of my preferred scheme: rename EdenStats to EdenThreadStats. (Make this diff easier to read by renaming ThreadLocalEdenStats to EdenStats in a separate diff.) In effect, implement the following naming scheme:
* **EdenThreadStats**: Statistics which are updated by one thread. (was EdenStats)
* **ThreadLocalEdenStats**: Statistics for all threads. Provides access to EdenThreadStats. (no change)
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D14822274
fbshipit-source-id: 236cd9878cd249a06d14e124050ee01329667a18
Summary: This diff makes `MononokeHttpBackingStore` uses `ServiceAddress` to manage remote address of the Mononoke API Server.
Reviewed By: chadaustin
Differential Revision: D14855661
fbshipit-source-id: 1c102f7b2c547dc6a5b5c2cba65a513313d90bdc
Summary: This diff adds `ServiceAddress` to utils. This is a class that encapsulates an address of a service that could be a traditional hostname & port pair as well as an SMC tier name. This eliminates the needs of the manually resolving network address for remote services.
Reviewed By: chadaustin
Differential Revision: D14845257
fbshipit-source-id: 9fe9847cca4bba0170be94b9c209247342708574
Summary:
Sometimes, EdenFS goes bonkers and talks to 'hg debugedenimporthelper' a lot for seemingly no reason.
Make these situations easier to debug by counting how many requests EdenFS makes to 'hg debugedenimporthelper'. These counters let us answer questions such as the following:
* When performance sucks, is EdenFS is making a lot of requests or only a few requests? I.e. is Hg slow at responding or is EdenFS very demanding?
* Does a recent performance issue correlate with EdenFS communicating with 'hg debugedenimporthelper'?
* Which engineers are outliers having orders of magnitude more 'hg debugedenimporthelper' requests than p50 engineers?
We could get fancier with these counters and include the number of bytes received, the duration of the request, etc. For now, just having a request count is useful.
Reviewed By: simpkins
Differential Revision: D14677339
fbshipit-source-id: 7f8f394fb0096aef65d6a8a45d7da5936db539a0
Summary:
D8559702 changed `folly::IOBuf::computeChainDataLength()` to return a `size_t`
Update our format specifier to match to avoid compiler warnings on Mac.
Reviewed By: chadaustin
Differential Revision: D14878220
fbshipit-source-id: 19e96bea07c57bb542a848b3688d65143db51d13
Summary:
The file descriptor API here needs to go away, so switch this API to NetworkSocket
It is expected that this commit will cause a number of Open Source projects to temporarily show up as broken. This is due to the fact that not all projects get synced to Github at the exact same time, so the builds may temporarily be fetching an older version of it's dependencies than it needs to :) It should fix itself quickly.
Reviewed By: yfeldblum
Differential Revision: D14673328
fbshipit-source-id: c5842fa5dc383d50043e0d8228e35d03b10a1c6b
Summary:
Ensure that the `TreeInode::createImpl()` code always releases the contents
lock at the end of the scope that was intended to signify the critical
section. Previously this was only unlocked when returning from this scope
successfully.
In particular, the old behavior could deadlock:
- We first created the new child inode.
- If saveOverlayDir() failed, we would throw an exception, causing us to not
release the directory lock yet.
- As we returned in the exception case, the local `inode` variable would be
destroyed before the `contents` argument, causing us to release the child
InodePtr's refcount before releasing the parent directory's lock. Because
we were the only user of the child inode we would end up needing to acquire
the parent inode's content's lock to check if it was unlinked.
This fixes the code to ensure that we always release the tree's contents lock
before the child inode becomes unreferenced.
Reviewed By: chadaustin
Differential Revision: D14848807
fbshipit-source-id: 28dcde842072b9ee1e7c63d54456d849e31af8fe
Summary: My editor configuration automatically format on save and I don't want to mix formatting with other meaningful changes.
Reviewed By: simpkins, strager
Differential Revision: D14859060
fbshipit-source-id: 8c906096b5c98a7f680bd2aeac9a7a5133da1ebc
Summary: Move Mononoke backing store initialization process to every request so we can adjust the configuration value without restarting the process.
Reviewed By: strager
Differential Revision: D14583357
fbshipit-source-id: 49ce2736229ce3062d34337757ebda6bb6eae16a
Summary: This is a helper function to help lazily initializing Mononoke backing stores only when necessary and release it when it is not used any more. See next diff in the stack for detail usage.
Reviewed By: chadaustin
Differential Revision: D14791528
fbshipit-source-id: c26811bc5c7aebcd02f704f10ad19bc35f8b9a21
Summary: The data structure is being modifed from multiple threads and should be synchronized.
Reviewed By: chadaustin
Differential Revision: D14712053
fbshipit-source-id: 25bbd45695895e6d85a6cc0c0d5321f9ef914d3f
Summary:
Remove the incorrect regex for `^4.*_fbk13`. From T34471852 this was
originally only intended to flag 4.11.3-52_fbk13 as bad. It is incorrectly
flagging `4.16.18-151_fbk13` as a bad release.
This regex isn't needed at all since that release is older than the
`minimum-kernel-version` setting anyway.
It really seems like this config setting should probably just be a single
regex, and shouldn't be split on `,` since you can just use `|` in the regex
instead. However that seems like a minor issue that should probably be
addressed in a separate diff.
Reviewed By: chadaustin
Differential Revision: D14815156
fbshipit-source-id: b00dd45cb212f6c3e33c02b6216c57d1510a233b
Summary:
We updating the encoding that mercurial uses for storing paths to UTF-8.
The current step is preventing files that have invalid names from being
committed. To that end we are updating the HgPrefetch test to submit
valid file names (valid UTF-8). This allows us to start using UTF-8
encoded string for the internal representation of file paths.
Doing encoding transformation between internal storage and system locale will
be implemented later. Normalization should also be handled at that point.
Reviewed By: simpkins
Differential Revision: D14812780
fbshipit-source-id: 1689910500391eed941df185ba92aa2d2f3f0960
Summary:
I want to use EdenStats in eden/fs/store/. EdenStats currently lives in eden/fs/fuse/, and making eden/fs/store/ depend upon eden/fs/fuse/ is confusing. (It's also confusing that some code in eden/fs/fuse/ is used on Windows.)
Reorganize the code: move EdenStats into eden/fs/tracing/.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D14677337
fbshipit-source-id: af26d214bcc3a9919920fbd4e59e6098fe4e3834
Summary:
This message was added in D14337058. It is logged at the `INFO` level, which
is enabled by default, but doesn't seem to add much value to normal production
logs.
Reviewed By: chadaustin
Differential Revision: D14712654
fbshipit-source-id: 5a86d883ace30e22d299046e33a6cd6247432857
Summary:
we need to be explicit about pulling in the dep for the
case where folly is not installed into a default installation
prefix.
Reviewed By: strager, pkaush
Differential Revision: D14683955
fbshipit-source-id: 0f302877674fb744ef8076641cd3fa72de74efe4
Summary:
the new getdeps script places the osxfuse/common dir into
its own install prefix, rather than dropping them into `external/osxfuse`.
This configures cmake to check in the installation prefixes known
to cmake.
Reviewed By: strager
Differential Revision: D14683956
fbshipit-source-id: 6e8a73341f8ddc21fef4b40b1f18a4a5128810e3
Summary:
`target_link_libraries` only allows passing things that
are libraries and expressly forbids passing in `-framework Foo`,
which is the sort of thing we get back from pkg-config on macos.
The result of misusing this is that cmake would add `["-framework", "-lFoo"]`
to the argv for the linker, which is totally broken.
Instead, we should use `target_link_options`.
Unfortunately, cmake seems to fail to do the right thing with the
` -framework CoreFoundation -framework Security` flags returned
from libgit2 on my system even using `target_link_options`; it somehow
ends up with a bare `Security` and fails to link. meh.
Reviewed By: strager
Differential Revision: D14680672
fbshipit-source-id: 62f65ddb4d07c8194cfc453cef1349b01be6c8b3
Summary:
This isn't a complete install but it is sufficient
to generate an `install` target for the getdeps build to run.
Reviewed By: strager
Differential Revision: D14680670
fbshipit-source-id: 9de1caa24c25702795842fe5b1b1f4d82aef24d8
Summary:
While testing out the new getdeps code I found that none
of the include directories from the probed libraries were being used.
The new getdeps installs each dep into its own prefix, whereas the
existing getdeps script installed them into the installation
prefix for eden itself. That meant that they were being implicitly
found from a single include directory.
In addition to this, I encountered linker failures for the pretty
printers; the solution to those was to add appropriate deps for
the modules that depend upon the pretty printers.
Reviewed By: pkaush
Differential Revision: D14638758
fbshipit-source-id: a4c2b4c79603c268e1b1c707a05c3cb0e3f2757b
Summary:
This function depends on things that are not available
when EDEN_HAVE_HG_TREEMANIFEST is not enabled.
Ensure that the entire function is disabled if that is the case.
Reviewed By: chadaustin
Differential Revision: D14638759
fbshipit-source-id: 3fe83b7b42357c2b818469214b016ff591058e35
Summary: Must have forgotten this when originally added.
Reviewed By: wez
Differential Revision: D14672454
fbshipit-source-id: 1702054e92ae6c88e974fc7d32e49835cca4bbbc
Summary:
The issue is that the compiler needs an `else` to see
that we can only reach the throw if none of the other paths are
taken; with that satisfied it believes that we are legitimately
constexpr.
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=67371
Reviewed By: chadaustin
Differential Revision: D14638234
fbshipit-source-id: f9524d2816580f41842a40e30118b03998c3660a