Summary:
Build the hg store on macOS with Buck. This primarily involves putting
ServiceRouter and Rust datapack support behind build switches.
Reviewed By: fanzeyi
Differential Revision: D17106241
fbshipit-source-id: da4643a11d6e5d0a6df23bff1d5e1454ab351abe
Summary:
The Eden macOS build has regressed in a few ways since it's been
broken due to RocksDB changes. Fix those small issues.
Reviewed By: fanzeyi
Differential Revision: D17107209
fbshipit-source-id: ef2e97a84e2403e7b7f36a4a01e2ceecab69106e
Summary:
Update the fsck code to save any orphaned symlink inodes that it finds as
symlinks in the repair archive directory, rather than saving the contents as a
regular file.
Reviewed By: wez
Differential Revision: D17170346
fbshipit-source-id: 4cba8b27233b728114a80a327ab519b039297aea
Summary:
Use the new `OverlayChecker` class to automatically scan for errors and
attempt to repair them if the overlay was not shutdown cleanly the previous
time it was used.
Reviewed By: wez
Differential Revision: D16596601
fbshipit-source-id: 9923565b101ba953e92909e502be6ef5895c5cbd
Summary:
Update the OverlayChecker's `inodes_` map to store `InodeInfo` objects
directly, rather than pointers to `InodeInfo` objects allocated on the heap.
This tweaks some of the error handling code to look up `InodeInfo` objects in
the map in a few places, to avoid storing raw pointers to `InodeInfo` objects.
The `InodeInfo` objects generally shouldn't move once we start performing
error checking, but it seems better to be conservative here and avoid using
raw pointers that are only loosely coupled with the original object's
lifetime.
This probably shouldn't really make much of a performance difference in
practice, since this code is likely to be disk I/O bound anyway. (It does
appear to make a small difference in performance for ASAN-enabled builds when
the inode files are warmed up in the kernel cache.)
Reviewed By: wez
Differential Revision: D16750736
fbshipit-source-id: b089a259cc83ffd2fda6f83b617d95ce4ac467f9
Summary:
We are probably going to add more inodemap counters, so add a prefix
now.
Reviewed By: fanzeyi
Differential Revision: D17142015
fbshipit-source-id: 4bd3cd4fd9234d8766864f364fef0b0d963f03b6
Summary:
This was causing problems on macos where various tools
would enumerate and helpfully try to preserve attributes across
copies. On macos this would result in appledouble metadata files
being created to track the metadata in the destination file,
which clutters up the repo and has surprising secondary effects
such as being picked up by glob operations in cmake build rules.
This diff simply stops enumerating the extended attribute.
Reviewed By: fanzeyi
Differential Revision: D17140414
fbshipit-source-id: 2924657dc75b900baf70595edfa72e5d0521a697
Summary:
This fixes a few issues with the library dependencies:
- The `eden_utils` library depends on `eden_service_thrift`, not
`eden_service`. By incorrectly depending on `eden_service` this introduced
a circular dependency which would cause a build failure, depending on which
order CMake chose to try and emit the link line.
- The `eden_config` library depends on code from `eden_model` (for `Hash` and
`ParentCommits`)
- The `eden_inodes` library depends on `eden_model_git` for the `GitIgnore`
logic. I also alphabetized the dependency list.
Reviewed By: wez
Differential Revision: D17124930
fbshipit-source-id: 70cbe81081fc1dc807cca13a93edc25ba270b01f
Summary:
Rename the `ThriftCppLibrary.cmake` file to `FBThriftCppLibrary.cmake`, and
also rename `add_thrift_cpp2_library()` to `add_fbthrift_cpp_library()`.
Explicitly calling this `fbthrift` helps clearly distinguish that this is
intended for use with fbthrift (https://github.com/facebook/fbthrift/), as
opposed to Apache thrift.
Reviewed By: wez
Differential Revision: D16738440
fbshipit-source-id: 9b255e06b71c98ad74a34989f564a211958dcdd5
Summary:
Implement the repair() logic for InodeDataError, MissingMaterializedInode, and
OrphanInode.
The most common errors we see after an unclean system reboot is an inode file
not being present at all (`MissingMaterializedInode`), or it being present but
empty (`InodeDataError`). When either of these errors occurs for a directory
it also produces `OrphanInode` errors, as all of the children entries in the
directory are no longer part of the directory hierarchy.
This implements repair logic for these three error types. The repair behavior
is largely similar to the Python version in `eden/cli/fsck.py`, with some
minor changes to the output paths in the `lost+found/` repair directory. This
also includes more unit test to better exercise handling when `InodeDataError`
and `MissingMaterializedInode` errors occur inside an orphan subdirectory that
needs to be extracted to `lost+found`
Reviewed By: strager
Differential Revision: D16577696
fbshipit-source-id: 948158c5a0c32b31574d93011281c42051645ad9
Summary:
If the hg_import_helper (really `hg debugedenimporthelper`) command
claims to not support treemanifest, then immediately fail.
Reviewed By: strager
Differential Revision: D15302513
fbshipit-source-id: 044a9cba877d65b6dab0ede6b431146a361d2801
Summary: Create config.toml in first Eden start up and store initial local store flag.
Reviewed By: chadaustin
Differential Revision: D16993074
fbshipit-source-id: dca8dbe60a892b7402ebf2f6c17df715dd2e8792
Summary:
This diff adds a fuse request timeout configuration setting
and threads it through to both the FuseChannel for our internal processing
and to the privhelper so that we can set an appropriate (slightly larger)
value for the kernel level daemon_timeout setting.
Reviewed By: chadaustin
Differential Revision: D16957552
fbshipit-source-id: a0fecc691d72914b5aebaed8a006dc0d6b0d7d12
Summary:
The macOS FUSE implementation has the concept of a daemon timeout,
which is used to set an upper bound on the length of time that the kernel will
wait for a request to be satisfied. If a request takes too long, the kernel
will shutdown the fuse device and the user is left with a relatively broken
experience in their repo until they restart the eden server. Critically, when
it is left in the broken state it is hard for the user to realize that they
need to restart the server because the kernel will still respond with EIO to
file accesses within the mount; even though it has been stopped, it isn't
fully unmounted until the fuse process stops.
This diff introduces a self imposed timeout in the fuse processing flow
and emits an ETIMEDOUT error when it is reached.
The intent is that we'll configure this timeout to be smaller than the
macOS daemon_timeout (which will be adjusted in a separate diff) so
that we don't trigger the problematic behavior when the kernel decides
that we've timedout.
This change is made for all fuse implementations, not just macOS, for
the sake of consistency: there's value to doing this on Linux as well,
to avoid some deadlock like scenarios: this should put an upper bound
on blocking in certain situations.
I've made the timeout default to 60 seconds, but haven't added any
configuration code for this yet; will do that in a follow on diff.
Reviewed By: chadaustin
Differential Revision: D16917954
fbshipit-source-id: 675539c43cf7f0009fd65d138081b9126464b7e0
Summary:
on linux kernel 5.1 we noticed that some caps were requested but not
known to the diagnostic code. This little diff adds a missing entry.
There is still one more that is unknown, but I think that we'll need to update
our copy of the fuse header to get that handled here.
Reviewed By: chadaustin
Differential Revision: D16953397
fbshipit-source-id: 30d19843a2cf1ec22fd469654773afaafa8e3b9b
Summary:
StartingGate was similar to std::barrier except that it guaranteed the
coordinating thread woke the worker threads. While this symmetry is
nice in concept, given enough threads, I doubt it pays for the
duplication.
Just adopt folly::test::Barrier (which is basically std::barrier)
instead.
Reviewed By: strager
Differential Revision: D16942136
fbshipit-source-id: 1691d8ef72b88c8867df74c8ed938f6c8d6ee094
Summary: Using a positive meaning rather than a double negative makes the build a tad simpler.
Reviewed By: wez
Differential Revision: D17000782
fbshipit-source-id: ef6c7b64708aa9b1f50c7ad4086c492a90c944f4
Summary: Using a positive meaning rather than a double negative makes the build a tad simpler.
Reviewed By: strager
Differential Revision: D17000620
fbshipit-source-id: ff27eb8098786b8ed6ed1ba81166b51e29e62d47
Summary:
The definition of RLockedPtr should use a non-const Subclass type, and not a
const one.
Reviewed By: ot
Differential Revision: D15356827
fbshipit-source-id: b8ad41e263f0e15ffa25b0698aa85eab8ca2ccb8
Summary:
The Java environment on Sandcastle macOS does not match corp laptops,
so allow disabling javadeprecated Thrift bindings on macOS for
now. Long-term, the better fix is to standardize the Java runtime and
JDK deployments across both environments so fbcode scripts can rely on
/usr/local/java-runtimes. See discussion in D16656570.
Reviewed By: strager
Differential Revision: D16676813
fbshipit-source-id: c21420bad554402c4c1e0881cabb962856719f5f
Summary: Allow building fuse_tester with open source gflags and glog.
Reviewed By: strager
Differential Revision: D16636221
fbshipit-source-id: 5128e936d5a6b6d0ed33bf34b770ee661728af3f
Summary: Now that StartingGate is in a more accessible dependency, use it in Eden's benchmarks.
Reviewed By: strager
Differential Revision: D16705853
fbshipit-source-id: 79ab9e9b32ab93f5b50a06a4889edb5e7b0d7e8b
Summary: On Linux and Mac, we provision .eden inside every directory, which contains symlinks to other locations. On Windows this won't work as well. First, Projected FS will cache all the .eden and will bump up the number of files on the disk. Second, it would be hard to simulate symlinks from Projected FS.
Reviewed By: strager
Differential Revision: D16724354
fbshipit-source-id: a560698fae8c7f14e475f1c1fb9ecc53e60c189f
Summary:
Implement OverlayChecker::computePath().
This also updates it to return a structure which can better represent
information if the inode is part of an orphaned subtree, which prevents us
from figuring out the full path to the root of the checkout.
Reviewed By: chadaustin
Differential Revision: D16577697
fbshipit-source-id: b543e2210fe8bc561d78194852962bf57ad9644a
Summary:
This ports much of the Python logic in `eden/cli/fsck.py` to C++, so that
the edenfs daemon can invoke it directly during mount initialization when
necessary.
This initial diff just contains the code to scan for errors, without any
repair logic.
This also includes some improvements to the logic for efficiency. For
instance, we compute less state during the initial scan phase. This makes the
scan phase faster, at the expense of making some of the repair steps slightly
more expensive if errors are found. This newer C++ scan logic is
significantly faster than the old Python version: scanning 260,000 inode files
took around 1m20 second for the Python code, and runs in under 3 seconds with
the new C++ code. (On an SSD with the files in question already warm in the
kernel cache.)
Reviewed By: strager
Differential Revision: D16577699
fbshipit-source-id: 2f8713a219ad882848cee0c57dca34f2c7808340
Summary: Redirecting the stdout and stderr to the eden log file. The log categories are not working as expected on Windows and will need to be fixed. At this point this doesn't log the "eden=DBG2" on Windows. One workaround for now is to remove "FOLLY_INIT_LOGGING_CONFIG()" to get the default logs.
Reviewed By: chadaustin
Differential Revision: D16653632
fbshipit-source-id: 028cdf1d698798ff3fb0284b3f0fa2df5570eaf0
Summary: Moving these functions to EdenInit so we could use them on Windows. This diff doesn't change the behavior.
Reviewed By: chadaustin
Differential Revision: D16653498
fbshipit-source-id: 5cc8c2f4b12f54035fb86ae20f3274245642d032
Summary:
Drop the `2` from the `fb303_thrift_cpp2` rule name.
The old `cpp` version of fbthrift is gone, `cpp2` is the only supported C++
version, so lets drop the `2` from this build rule name.
I plan to change some of the CMake thrift rule generation code soon, to
support multiple languages (namely Python). As part of this I plan to
automatically add a language-specific suffix(`_py` or `_cpp`) to the build
rules. It seems better to use `_cpp` rather than `_cpp2` for C++ libraries.
Changing the name for this fb303 rule will make it fit this planned naming
scheme.
Reviewed By: chadaustin
Differential Revision: D16653264
fbshipit-source-id: e4da70c79ff14e9981717ab909d982770d104512
Summary:
Now that we have unsafe access to the contents of a
folly::Synchronized, InodeMap::updateOverlayForUnload can just access
the storage because it knows nobody else has access.
Reviewed By: simpkins
Differential Revision: D16632778
fbshipit-source-id: 4b2c1d6679e2655671d43899407bac1a19f2cab9
Summary:
`StartupLogger::exitUnsuccessfully()` is declared as `noreturn`, and its last
statement is to call another `noreturn` method. However, this other method is
virtual, and the behavior of the `noreturn` attribute on virtual functions
doesn't seem consistently implemented across compilers. gcc doesn't seem to
treat the virtual method as `noreturn`, and therefore still warns that
`exitUnsuccessfully()` may return.
This suppresses that warning by adding a `folly::assume_unreachable()`
statement.
Reviewed By: fanzeyi
Differential Revision: D16653184
fbshipit-source-id: e0a2d8a4420d3582f849e93568a5081827a788f5
Summary:
The toHash field is unnecessary in HashUpdateDeltas since we only ever iterate in reverse. We instead keep track of the current hash in the journal itself instead of looking at the toHash of the latest HashUpdateDelta.
Delta Struct Sizes:
> File Change: 88
> Hash Update: 96
Reviewed By: strager
Differential Revision: D16522519
fbshipit-source-id: 43baccc8ef2579f72609cc84e81e218794b11725
Summary: Create the second delta type HashUpdateJournalDelta that keeps track of changes to the hash [and the unclean files associated with that hash change]. Journal methods were updated to account for the different delta types.
Reviewed By: strager
Differential Revision: D16520444
fbshipit-source-id: 2a5cea11c9e70e30f6db55d9c8e33f9322ae91fc
Summary: Add in two tests for deltas that change commit hashes since a test for this did not exist before. Previously a bug in setting the fromHash of a result only showed up as a failure in Watchman's integration tests with Eden.
Reviewed By: chadaustin
Differential Revision: D16528248
fbshipit-source-id: 56eede749ef2da4dc492a1f7376dc07ca8aa3050
Summary: Deltas are stores in a deque and therefore not malloced independently so therefore there is no reason to be getting the goodMallocSize of them. We also need to do accounting for the buckets in the deque that the deltas fall into.
Reviewed By: strager
Differential Revision: D16566675
fbshipit-source-id: 4506fbbcc2044b8fdfe6244313ef7480cfa5151e
Summary: Record a rolling sum of the time taken by any FUSE call on a per-process basis
Reviewed By: strager
Differential Revision: D16553149
fbshipit-source-id: 54f1e453916727a40f245b294239dc1b232a8967
Summary:
Change the behavior of `newEdenError()` to simply join its message arguments
together using `folly::to<std::string>()` rather than using
`folly::sformat()`.
Only a couple locations in the code were actually passing in a format string
correctly. Several other locations in the code were incorrectly passing in
arguments that they expected to be joined. This resulted in the error
messages being truncated, as the initial string did not contain any format
placeholders, so the trailing arguments were simply ignored.
Reviewed By: strager
Differential Revision: D16592709
fbshipit-source-id: d56f5387e05da2a68d94e6badac28ffcf68d6eab
Summary: Uses the existing RequestData class to make calls to static functions to set and get the `didImportFromBackingStore` flag.
Reviewed By: simpkins
Differential Revision: D16461868
fbshipit-source-id: e3ed39249f5dd1a842ad06a204b5933014b12f7f
Summary: hg debugedenimporthelper should either have the repo path as argument or the current working directory should be inside a repo. Setting the current working directory for the process.
Reviewed By: strager
Differential Revision: D16565042
fbshipit-source-id: d6e826bebf58fd58f5250e6c477849ca147628c8
Summary:
Update `EdenServer::mount()` to correctly handle errors that occur during the
mount `INITIALIZING` phase. Previously the code did not add error callbacks
to the `Future` result to handle errors during initialization. As a result we
would propagate the exception back to the thrift caller, but the `EdenMount`
object would remain in our mount point list, stuck forever in the
`INITIALIZING` state.
Reviewed By: strager
Differential Revision: D16590032
fbshipit-source-id: 9adbdf05441dad815096b195ece36f3d958c96a9
Summary:
Update `FsOverlay::validateHeader()` to throw `EdenError` exceptions rather
than `std::system_error`. These exceptions are generated when we find invalid
data, rather than from system calls that return errno values. Previously the
code was using `folly::throwSystemErrorExplicit()` and
`folly:throwSystemError()`, and passing in a made-up `EIO` value.
However, the latter two calls were incorrectly using
`folly::throwSystemError()` rather than `throwSystemErrorExplicit()`. As a
result the `EIO` parameter was being treated as part of the error message, and
the code was appending the error description for whatever value `errno`
happened to be set to at the moment.
This fixes the code to just throw `EdenError` exception types, since these
errors aren't caused by a real `errno` value.
Reviewed By: strager
Differential Revision: D16577698
fbshipit-source-id: 9487c34f04da99d397611f005f00f02114b12094
Summary:
Fix `FsOverlay::initOverlay()` to also create the `tmp` subdirectory, which is
required for the `FsOverlay` code to function properly. Previously this
directory was created by the `Overlay` class. However, `FsOverlay` should be
responsible for creating it so it can be used from other locations besides
just `Overlay`.
I also removed the code that checks if the `tmp` directory needs to be created
on restart. The code to create `tmp` has been present for over a year
(D8330070), and we no longer have anyone using EdenFS versions older than
this.
Reviewed By: strager
Differential Revision: D16577700
fbshipit-source-id: 3eaa28003159a03084c4f367c7b96d69fc1702d0
Summary:
There is no point to call `readBE<uint64_t>()` here. The only way these calls
can fail is if the buffer is too short for the data, but we already checked
the buffer length at the top of the function.
Reviewed By: strager
Differential Revision: D16577701
fbshipit-source-id: 8b57cec05453b9b7224470671e032e3aefe41033
Summary: Replace the uses of std::unique_ptr<JournalDelta> with just the JournalDelta itself to avoid an extra allocation.
Reviewed By: chadaustin, strager
Differential Revision: D16572089
fbshipit-source-id: be080b2fb9096f6c8783e2ecae21a99466336f6f
Summary:
Adds a debug command to eden such that users can flush the journal and cause the subscribers to get a truncated result, this will be useful for debugging [to debug we won't have to lower the memory limit and repeatedly touching files to fill up the journal].
Can be used with "eden debug flush_journal"
Reviewed By: chadaustin
Differential Revision: D16348811
fbshipit-source-id: fdbe6729d0393c424addcd42a091de440383035b
Summary: Refactoring the addDelta function to allow other Journal functions to add deltas while holding the deltaState lock. This refactoring will be used to create the flush function for the journal (which needs to hold the lock to empty the journal but wants to add a delta before releasing the lock).
Reviewed By: chadaustin
Differential Revision: D16341196
fbshipit-source-id: 7b7b1d933802b466efe624378206c72c71469129
Summary: Delete tests that rely on flatmanifest support in Mercurial.
Reviewed By: strager
Differential Revision: D15302489
fbshipit-source-id: 7e46f21bfe3ae910e205e73bca95d9a33f4a6816
Summary: D16461081 broke the windows build of Eden by not updating the Windows mount code, this diff makes the same change as that diff in the Windows portion of the code.
Reviewed By: chadaustin
Differential Revision: D16549392
fbshipit-source-id: eee1bede08c1b27329b026746dda6863519ed10b
Summary: Keep track of the longest query we had to resolve in accumulateRange. This stat will be helpful in determining how large the memory limit of the journal should be since we will know how far back people usually need to go.
Reviewed By: strager
Differential Revision: D16227920
fbshipit-source-id: a41c3b9f16b701cd8165e20409888983b8899dab
Summary: The journal will now keep of how many reads from accumulateRange have been truncated. This is a useful metric that will allow us to see how often we are forcing Watchman to use its fallback of creating a fresh instance and help us in calibrating if we find our memory limit is too small.
Reviewed By: strager
Differential Revision: D16017968
fbshipit-source-id: 95f4fbd1fd2d8523ff397202172408e1c89669be
Summary: D16096960 accidentally caused the truncation code to be run twice (once inline and once via a function code), this shouldn't cause any difference in outcome but is unnecessary.
Reviewed By: strager
Differential Revision: D16508798
fbshipit-source-id: 12781aee98e70e5105c5476d29cf5cdd1e31062d
Summary:
D15528156 inadvertently changed the return type of the `getPid()` call from
`i64` to `i32`. This is a backwards-incompatible change, and causes new
clients to reject the 64-bit response from older EdenFS daemons. This breaks
`edenfsctl status`, `edenfsctl clone` and other commands.
Reviewed By: pkaush
Differential Revision: D16522765
fbshipit-source-id: eecd344ee4b963d638576f146a87fc88a5003e55
Summary: Set up the infrastructure to add in Timeseries to the journal so that we can add in stats for the journal that relate to the whole process. For example, allow us to add in a truncatedRead TimeSeries easily as done in D16017968
Reviewed By: chadaustin
Differential Revision: D16461081
fbshipit-source-id: 964ff32e62aed0369da434793491b857c136b074
Summary: Now that Eden depends on open source fb303, EDEN_HAVE_STATS is unnecessary. Remove it.
Reviewed By: simpkins, strager
Differential Revision: D15526905
fbshipit-source-id: 2354f1b92545a089de0e91e7c33515fa0b74b067
Summary: Add a dependency from the eden open source build to the fb303 open source build and switch EdenServiceHandler to BaseService.
Reviewed By: simpkins
Differential Revision: D15528156
fbshipit-source-id: 2ca5c31dd9fcc9bac43fd399b27f33b6f2c5ebfc
Summary:
From andrewjcg:
> Basically, rocksdb headers exist at both e.g. rocksdb/src/include/rocksdb/flush_block_policy.h and rocksdb//flush_block_policy.h. The latter are never meant to be used, but Apple platforms by default add the repo root to the include dir before anything else. This means depending on file-relative inclusion from within rocksdb, we end up getting different versions of the header, which busts through #pragma once causing multiple definition errors.
This is split out from D15125826.
Reviewed By: andrewjcg
Differential Revision: D16353321
fbshipit-source-id: 095e3b7fb4f56b0be42d18fe4009336b65c8eb5e
Summary:
To save on memory the journal will now compact the same action repeated multiple times into the same action. This means that modifying the same file 100 times in a row results in 1 Journal delta instead of 100. [The results will cause Watchman to act the same since all queries are down from the current time, changes should only be visible by the number of deltas in the journal, how much memory the deltas are using, "eden debug journal" which will show that sequenceID's were skipped, and the fromSequence/fromTime returned by accumulateRange might be different]
**Memory Improvements:**
For buck commands, 1 run was conducted for each with a buck clean done before each build and then eden being restarted (so the clean did not affect the outcome) [results are formatted as 'with compaction' / 'without compaction']
“buck build mode/opt eden”
Entries: 154145 / 206108 [25.2% reduction]
Memory: 46.2 MB / 61.4 MB [24.7% reduction]
“buck build mode/opt warm_storage/common/...”
Entries: 318820 / 405016 [21.3% reduction]
Memory: 95.8 MB / 121.5 MB [21.2% reduction]
For Nuclide the result was calculated by getting the number of entries in the journal vs the last sequence ID in the journal ('entries we actually have' / 'entries we would have without compaction')
Using Nuclide’s Smart Log and Checking Out various commits / arc pulling:
Entries: 6091 / 23671 [74.3% reduction]
Reviewed By: chadaustin
Differential Revision: D16096960
fbshipit-source-id: f542ae32c889ebc9da442285d808ce75247f7e65
Summary:
This diff updates Eden's journal to be bounded in terms of memory usage which should help lessen the likelihood of Eden OOMing and taking up a large amount of our users' resources.
The memory limit is set to be 1 GB per journal [so a user with 3 mounts could expect the journals to possibly use up to 3 GB of memory].
The landing of this diff will need to wait until a version of Watchman that can handle truncation is deployed on all machines using Eden. This means we need to wait for a version of Watchman with D16219267 to fully land to machines.
Reviewed By: strager
Differential Revision: D15954994
fbshipit-source-id: 9a6425527f10a1ce051feb8fc7d092a84712f338
Summary:
We install the python files under `eden/fs/service` in a package named
`eden/thrift` in the built python binaries that use these modules. This
moves the source files into an `eden/thrift` subdirectory so that the source
directory layout more closely matches the final binary layout.
This will make it easier to run several of Eden's Python-based tools directly
from the source tree, without having to do as much directory layout
transformation. This is particularly helpful on platforms like Windows, which
don't currently have an equivalent of "live PARs" which can be run from the
source tree without requiring a rebuild after each file edit.
Reviewed By: chadaustin
Differential Revision: D16354622
fbshipit-source-id: 4b58cc96451b1ee5441714aaf74f5e3b6ada9eaa
Summary:
Let's minimize the amount of work we do before dropping privileges.
This shouldn't matter too much, since it is just a memory allocation and copy,
but I think we should keep the `dropPrivileges()` call as the very first thing
in `main()` to help prevent people from adding new functionaliy before it in
the future.
This also adds some block comments to help encourage people to only put code
after the "Root privileged dropped" comment moving forwards.
Reviewed By: chadaustin
Differential Revision: D16371783
fbshipit-source-id: 1ce5115bfa71565f0fe12ac9c9442bddb771ed48
Summary: In a previous diff we added the support to pass the command line to EdenServer but missed from the Windows main(). Until we unify the main function we need to add it at two places.
Reviewed By: simpkins
Differential Revision: D16370228
fbshipit-source-id: 47b9842fbca709e1d3c832db3d82765f2e5f8930
Summary:
I made the mistake of migrating the Thrift API for edenfs process
checks without a fallback for older edenfs running processes.
If getDaemonInfo() is not available, fall back on getPid().
Reviewed By: strager, pkaush
Differential Revision: D16365848
fbshipit-source-id: aee5c43c8c40db9a38fef7ca601f7639daa4adb8
Summary:
Open source fb303 will not have getPid() or getCommandLine(), so
introduce a new method for Eden's tests.
Reviewed By: fanzeyi
Differential Revision: D16292993
fbshipit-source-id: 5cdc006ec0ee15f50a3e1cebe9b46a3ea275ff78
Summary: I found out that the journal stats callbacks that were getting registered were not getting unregistered, this diff fixes that.
Reviewed By: chadaustin
Differential Revision: D16187569
fbshipit-source-id: 8c84e1515e376ccd7036a22c06e2e6b98dc62342
Summary: Updating Eden to send its blob cache memory usage to ODS. This will be useful in seeing how much of our overall memory usage is coming from the Blob Cache and debugging what is using large portions of Eden's memory.
Reviewed By: chadaustin
Differential Revision: D16157709
fbshipit-source-id: abae8e8da22369c7757187f998a1f6bef31d74e9
Summary: Running `eden debug journal -f` will print and follow the eden journal in a similar style to the unix `tail -f` command.
Reviewed By: chadaustin
Differential Revision: D16112458
fbshipit-source-id: 5304cd0f857bdbeca41c2591e98920f4f1fc8f42
Summary: Sets up a thrift interface to set the size of the journal (until truncation is added in the size field in the journal currently does nothing other than being viewable from getMemoryLimit)
Reviewed By: chadaustin
Differential Revision: D16042286
fbshipit-source-id: bc0acdf4ac5516cfac66fa0fbd87254d08ad479b
Summary: Added the cli command `eden stats object-store` for querying the counts on what part of the object store was responsible for finding the blob or blob size (local store or backing store). This will tell us how well local and in-memory caching works for different workflows.
Reviewed By: chadaustin
Differential Revision: D15934535
fbshipit-source-id: 70345f11a51c3c6996dc001d4101744395a3d182
Summary: eden_win_mount, eden_win_utils and eden_win_store include journal which depend on streamingeden_thrift. Adding the dependency to fix the build order.
Reviewed By: fanzeyi
Differential Revision: D16065869
fbshipit-source-id: d3bd5887e6c675885d284671bef7e514515d5fc0
Summary: Journal not specifying template argument for optional was causing eden builds to fail for old versions of C++. See D15978960.
Reviewed By: chadaustin
Differential Revision: D16059722
fbshipit-source-id: fb9903e304d9c5f79fc2d34f7763c3ada6ae4553
Summary: Create a counter for the end-to-end duration of the journal.
Reviewed By: chadaustin
Differential Revision: D15993675
fbshipit-source-id: 260a58a7b37dfa16b82b3df187ad7f7263ff67cf
Summary: Shows the end-to-end duration of the journal in "eden stats"
Reviewed By: chadaustin
Differential Revision: D15993261
fbshipit-source-id: 46471faca17d4f12ccdd8cea55b2722e33519a74
Summary:
Moving from the previous linked list approach to having a deque in the Journal.
The memory used by the deque is not tracked currently, the difference might be negligible though. I ran a script that touched a file 62 million times and here is the result comparing rss_memory used in eden and the journal's estimated memory: https://fburl.com/ods/6sq8g5vc and it still seems to closely estimate the right amount.
Reviewed By: strager
Differential Revision: D15944614
fbshipit-source-id: 6a1ac34ecd80c0eecb80411984f88f62ae712e91
Summary: Change getLatest such that it just returns Info about the latest delta and not the delta itself.
Reviewed By: strager
Differential Revision: D15931214
fbshipit-source-id: c7a1cf4d62cdd4f9396fab46354eabcbb31f32c0
Summary: Combine the backend function that iterates over the deltas into one function so we do not need to modify the code in multiple places as we start to modify the internal structure (Additionally pulled the one use of the code outside of the journal into the journal)
Reviewed By: strager
Differential Revision: D15912907
fbshipit-source-id: 2f59ff9e7f33ffa5b420859153a609e68bda10b4
Summary: Prior to this diff "eden debug journal" got the latest journal entry and manually seeked for the one "limit" prior to it, this has been updated to just passing the limit to the journal. [This will make the queries always happen from the tip of the journal, since in theory currently something can be added to the journal after the python script gets the latest delta]
Reviewed By: strager
Differential Revision: D15972018
fbshipit-source-id: 0ee0dd88a1e9edef5ccce3b3da2dbc09aa64f8a9
Summary:
Implements size-only local storage, as opposed to storing metadata. This is useful when the blob's SHA-1 is not needed. This diff prevents SHA-1 computations, which can be especially expensive for large blobs.
From D15934535, operations such as `ls -l` and `stat` will get the size of a blob in two ways:
1) The blob's size is already stored locally, so it will be deserialized and returned.
2) The blob is fetched from the backing store, stored, and its size is returned.
This diff optimizes the second case, because SHA-1 is no longer computed.
Reviewed By: strager
Differential Revision: D15723239
fbshipit-source-id: a868f3bf6b68a83ddafb057dc3e4e65f0a2dd989
Summary:
These allow the cli to setup and tear down mounts and
have the eden server keep track of them.
Reviewed By: strager
Differential Revision: D15707318
fbshipit-source-id: abdb8eaa28c8c67c8211a8af1647efe3a083e998
Summary: We need --edenDir support to run muliple instance of Edenfs, which is required to run the integration tests.
Reviewed By: simpkins
Differential Revision: D15951483
fbshipit-source-id: a516159cdeb5f046f795fc28399a2af5fe8a9f95
Summary: Adding a nullptr check for the case when the journal is empty and removing the use of default parameters.
Reviewed By: strager
Differential Revision: D15907329
fbshipit-source-id: 787b4a44f835fd8d128496ee6655e02987db98a7
Summary:
This contains several fixes to LocalStore handling during shutdown.
- Have EdenServer explicitly call localStore_->close() during shutdown.
This ensures that the local store really gets close, just in case some other
part of the code somehow still has an outstanding shared_ptr reference to
it.
- Add synchronization around internal access to the RocksDB object in
RocksDbLocalStore. This ensures that calling `close()` is safe even if
there happens to still be some outstanding I/O operations. In particular
this helps ensure that if background GC operation is in progress that
`close()` will wait until it completes before destroying the DB object.
This also improves the code so that calling subsequent methods on a closed
RocksDbLocalStore throws an exception, instead of simply crashing.
I don't believe the additional synchronization in RocksDbLocalStore should
have much impact on performance: the synchronization overhead should be very
low compared to the cost of the RocksDB reads/writes.
Ideally some of this synchronization logic should perhaps be moved into the
base `LocalStore` class: all of the different `LocalStore` implementations
should ideally ensure that `close()` is thread-safe and blocks until other
pending I/O operations are complete. However, that requires a bigger
refactoring. I may attempt that in a subsequent diff, but for now I mainly
want to address this problem just for RocksDbLocalStore.
Reviewed By: strager
Differential Revision: D15948382
fbshipit-source-id: 96d633ac0879b3321f596224907fcfe72691b3f0
Summary: This diff takes care of importing blob from Mononoke and Mercurial at the same time, also improves the name situation in the statistics counters.
Reviewed By: strager
Differential Revision: D15768557
fbshipit-source-id: 10cf831b1ae6dc9e6b91f1e96508c4fa92583743
Summary:
watchman/eden have been using the new Thrift streaming for nearly a year now and are the last users of the old
Thrift streaming. This diff kills the fallback `legacySubscribe` method, which complete's migration to the new Thrift
streaming and unblocks the complete removal of wangle-based streaming from Thrift and fbcode.
Reviewed By: wez
Differential Revision: D15878892
fbshipit-source-id: ec0d270dba79e56c7e41afbf36669a08e5a15518
Summary: This code calls close on fd while exiting. If the file doesn't exist the open will return -1 and close (-1) will be called.
Reviewed By: strager
Differential Revision: D15951485
fbshipit-source-id: ea3a52517847d75e9a822e51f360be7cb2c411da
Summary: Disabling this task because it's not implemented and will throw and crash the fs.
Reviewed By: strager
Differential Revision: D15951484
fbshipit-source-id: 0819219d621edfad488707b709705f636358ede8
Summary:
D15901794 deleted the move `ThriftLogHelper` move constructor, to suppress
some warnings from clang on Mac OS. However, this causes build failures on
Windows, which needs the move constructor. Clang does not need the move
constructor due to copy elision, which is guaranteed by C++17.
This also changes the itcLogger_ member variable to be a copy instead of a
reference: `folly::Logger` is just a simple wrapper around a
`folly::LogCategory*`, so making a copy should be just as efficient than using
a reference (if not more, since we no longer have to traverse the reference).
Reviewed By: pkaush
Differential Revision: D15936517
fbshipit-source-id: 92ab47ac9985a04e360f529bc99705a7cad5c5b2
Summary:
If edenfs was started using `sudo`, the `$USER` environment variable will be
set to `root` rather than the actual user. When we drop privileges make sure
we restore the value of `$USER` as well.
The `$USER` variable isn't checked anywhere else in edenfs itself, but it
matters for subprocesses we spawn, like `hg debugedenimporthelper`.
I also changed the code to clear the `SUDO_*` variables as well, mostly
just for good measure.
Reviewed By: kulshrax
Differential Revision: D15929539
fbshipit-source-id: e022c7ae762e2a5e86d0227058bb476aff17cf55
Summary:
Add a periodic task for performing LocalStore management tasks. For now only
the RocksDBLocalStore class implements this management task.
When this periodic task runs the RocksDBLocalStore object computes how much
space each of the column families are using and publishes this as fb303
counters. If the total size of the ephemeral column families exceeds a
configurable limit it then triggers a background garbage collection task.
I also added a new `edenfsctl stats local_store` command that reports the new
counters added by this diff.
Reviewed By: chadaustin, strager
Differential Revision: D15798505
fbshipit-source-id: 25ca4ba80f5a9c4a1a09dc08633c7b3af363d7ff
Summary:
This diff makes eden to import tree from Mercurial and Mononoke at the same time.
Since both futures will finish even if the other one finishes early, this change will make sure:
1. Mercurial fills its cache
2. User gets the fastest experience
3. Content SHA1 and file size still get fetched from Mononoke API Server since we store metadata in a separate space:
Reviewed By: chadaustin
Differential Revision: D15768555
fbshipit-source-id: 5407b5e1e9b0ea6c90905de5adcdfbbf09aa6469
Summary: This diff clears the use of `LocalStore::WriteBatch` in `HgBackingStore` so operations does not share the same write batch. Sharing write batch is causing segfault when we switch to importing concurrently from Mercurial and Mononoke.
Reviewed By: chadaustin
Differential Revision: D15768556
fbshipit-source-id: 1b0baee3a561d79b55d455c1dcb0d66475a1ea8d
Summary: EdenMount.cpp includes folly/Subprocess.h, but uses none of the included symbols. Drop the unnecessary #include.
Reviewed By: simpkins
Differential Revision: D15773136
fbshipit-source-id: 4f30af9a1afcc3258aaa00d519d2aceeda72824f
Summary: Update the license headers in the remaining Eden files.
Reviewed By: wez
Differential Revision: D15487083
fbshipit-source-id: cea9cc133907eadf5afc069f5d420f686b4c1886
Summary:
Update the copyright & license headers in thrift files to reflect the
relicensing to GPLv2+
Reviewed By: wez
Differential Revision: D15487082
fbshipit-source-id: 33f68617037f36c07075fb962a16a4d8f55bd6a6
Summary:
Update the copyright & license headers in CMake files to reflect the
relicensing to GPLv2
Reviewed By: wez
Differential Revision: D15487079
fbshipit-source-id: 715e559464c19a0070d6e55a095b3fc7d61ad2f8
Summary:
Update the copyright & license headers in Python files to reflect the
relicensing to GPLv2
Reviewed By: wez
Differential Revision: D15487088
fbshipit-source-id: 9f2138dff41048d2c35f15e09a04ae5a9c9c80dd
Summary:
Update the copyright & license headers in C++ files to reflect the
relicensing to GPLv2
Reviewed By: wez
Differential Revision: D15487078
fbshipit-source-id: 19f24c933a64ecad0d3a692d0f8d2a38b4194b1d
Summary: Merge was a function on a JournalDelta that created a new JournalDelta and optionally left it connected to old JournalDeltas. AccumulateRange is a new function on the Journal itself (acting on the latest delta) that creates a JournalDeltaSum (which can't be connected to older Deltas)
Reviewed By: chadaustin
Differential Revision: D15881452
fbshipit-source-id: 573505c1171f78d46afc98f1db9b5b9ee2fff60f
Summary: Making addDelta private and giving users a more user-friendly way of appending entries to the journal.
Reviewed By: chadaustin, strager
Differential Revision: D15868089
fbshipit-source-id: 00c8a3066f0e4483e3c792651ade5f6a7ea05eed
Summary:
Some newer versions of `clang` (such as Apple's version 11) will warn/error out if a constructor or assignment operator
marked `default` is implicitly deleted (e.g., if the object contains a non-moveable/non-copyable member). This diff
removes all such defaulted constructors/assignment operators, which I ran into while building `edenfs` on my Macbook Pro.
Reviewed By: chadaustin, strager
Differential Revision: D15901794
fbshipit-source-id: 794ed8377693a6735bb567635dc919bc678751a4
Summary: This diff allows watchman to connect to `EdenServer` over an rsocket stream.
Reviewed By: simpkins, strager
Differential Revision: D15900246
fbshipit-source-id: e4a99fcd7b9c75b6ad2c331c1bbf13d22f78cfd7
Summary:
Update `RocksDbLocalStore::getApproximateSize()` to use `DB::GetIntProperty()`
rather than `DB::GetApproximateSizes()`.
The `GetIntProperty()` call reports properties about the overall column
family, as opposed to `GetApproximateSizes()` which supports querying size
information for specific key ranges. We always want data about the full
table, so the `GetIntProperty()` call seems to make more sense for us.
Also change the code to include both the SST file size (which we were querying
before) and the memtable size (which was not included before). The
`GetApproximateSizes()` call allow this to be queried too, but using
`GetIntProperty()` still seems better since we don't care about a specific key
range.
Reviewed By: wez
Differential Revision: D15798506
fbshipit-source-id: 86d087876d838c0ff4ae517ccac2e8b8bf31e304
Summary:
Add a debug log message at the end of each PeriodicTask iteration, reporting
how long the task took.
Reviewed By: wez
Differential Revision: D15798504
fbshipit-source-id: 3f33877cb4a9ddb8d966877b005e6d4c60d7dbe0
Summary:
Add a periodic task to reload the configuration file from disk. By default
this runs once every 5 minutes, but this interval can be controlled from the
config file.
At the moment reloading the config file does not do much other than update the
interval for how frequently the config file is reloaded. However, I plan to
add additional periodic tasks shortly that are controlled by this config
setting.
This will also make it possible for other parts of the code to
access the config settings in the `ServerState` and use them as-is without
checking to see if they reloaded. Currently all of the code that accesses
config values performs a check to see if the config needs to be reloaded. If
we want to switch to Mercurial-style configs in the future that check will be
substantially more expensive.
This diff also includes a new thrift call to force the config file to be
reloaded immediately. This can be used to restart automatic config reloading
if it is ever disabled in the config file.
Reviewed By: wez
Differential Revision: D15756357
fbshipit-source-id: 1999f4730903633ce838842932a6ae6a65eda4e6
Summary:
Migrate the code from accessing optional Thrift fields directly to a safer
`optional_field_ref` API. See https://fburl.com/safe for more details.
The output of this codemod has been reviewed in D13259011.
To preserve semantics, each unchecked access is replaced with an explicit call
to `value_unchecked()`. If you are sure that accessing a field is safe (the
field is marked as set), you can later replace `value_unchecked()` with
`value()` or dereferencing (`operator *`):
```
ThriftStruct s = ...
- auto foo = s.foo_ref().value_unchecked();
+ auto foo = *s.foo_ref(); // will throw if s.foo is unset
```
drop-conflicts
Reviewed By: stevegury
Differential Revision: D15786632
fbshipit-source-id: 7c3f089dfa79597b189da6996cb76dcdbd5ed612
Summary: The first step towards decoupling size from the metadata object in the local store. This is necessary for the next step, which is to serialize size separately from the combined metadata object.
Reviewed By: chadaustin
Differential Revision: D15719637
fbshipit-source-id: 979555e3e362f2b698b3af3a0b6db1e350e07dcd
Summary: Wrote tests for retrieving hashes from the object store and refactored existing tests. Includes positive and negative tests.
Reviewed By: strager, pkaush
Differential Revision: D15697488
fbshipit-source-id: 44aa8a36fbc1d1e56dcbbb6bcb665d3784cbb476
Summary: `getSize` and `getSha1` were misleading function names, since the functions refer to the size and hash of a given blob and not the object store itself. These functions have been renamed to `getBlobSize` and `getBlobSha1`.
Reviewed By: chadaustin
Differential Revision: D15696510
fbshipit-source-id: 4dd31659f60969fa90d8e2b39f43c46a2b7dff7c
Summary: I decoupled the getSize() function from the getMetadata() function, using a naive implementation for now. This was necessary because I want to add support for fetching only the size of a blob during a request like `ls -l`. Right now, the size and hash of a blob are coupled in a Metadata object, so if the size is requested, the whole file must be downloaded to calculate the hash, which is expensive for large files.
Reviewed By: chadaustin, strager
Differential Revision: D15678216
fbshipit-source-id: 8f68692768faaae0e65373ffe608d09ae49bbc42
Summary: HgCommand.h and related code isn't used anywhere. Delete it.
Reviewed By: simpkins
Differential Revision: D15773223
fbshipit-source-id: c4c116150bb74e29e4f220d3b0cbf988d4a1b1c5
Summary:
The eden/fs/store library depends on eden/fs/config. The changes in
D15428932 exposed a build failure due to this missing dependency.
Reviewed By: strager
Differential Revision: D15768197
fbshipit-source-id: 4482e3a9027fd49163af39d6887da709be810b0a
Summary:
The `FieldConverter<bool>` code previously relied on `folly::to<string>()` for
all arithmetic types, including `bool`. Unfortunately `folly::to<string>()`
returns `"1"` and `"0"` for booleans, which isn't terribly human-friendly.
This updates the code to explicitly return `"true"` and `"false"` as the
output for converting boolean values to strings.
Reviewed By: strager
Differential Revision: D15550014
fbshipit-source-id: 69a3385e529e2940a8be20dbcbeda92e5836d969
Summary:
Add a `FieldConvertor` implementation allowing a config field to be parsed
into a `std::chrono::nanoseconds` value.
Reviewed By: strager
Differential Revision: D15428932
fbshipit-source-id: d3ace45a2986717c798897534e7b3015eda35fa9
Summary:
The `ConfigSetting` code previously used `folly::to<string>()` to convert
field values back to strings for debugging. However, this only works for
values that actually are supported by `folly::to()`
This changes the code to use the `FieldConverter` to perform the conversion
back to string. The `FieldConverter` class now contains the logic for doing
conversion both from strings and back to strings.
Reviewed By: strager
Differential Revision: D15428933
fbshipit-source-id: a37a978d151bf0ba79ca2a2b9107401c847b6591
Summary:
Add a `durationToString()` function. This converts a duration value to a
string that can be parsed back with `stringToDuration()`.
Reviewed By: wez, strager
Differential Revision: D15428935
fbshipit-source-id: 2840663385434bc35f04c16d2570fd86e234cb3f
Summary:
Add a `stringToDuration()` function to parse a string into a
`std::chrono::nanoseconds` value.
Strings are parsed as a sequence of `[number] [unit]` pairs.
For instance:
`1m30s` --> 90 seconds
`5s10ms` --> 5010 milliseconds
`1d3h10m` --> 1 day, 3 hours, 10 minutes (1630 minutes total)
Reviewed By: strager
Differential Revision: D15428934
fbshipit-source-id: 63f6614595c1aaa08c0d03633c6d1b53ca5bf3d5
Summary:
Update the `FieldConverter` code to support parsing any arithmetic type
(integers and floating point) using `folly::to<T>()`. Also change the `bool`
conversion implementation to use the same code.
This does change the `bool` implementation to accept any string accepted by
`folly::to<bool>(string)`. The old code only allowed `true` and `false`. The
new code allows other strings: `yes`, `no`, `y`, `n`, `0`, `1`, `on`, `off`
This also makes the implementation more efficient by eliminating some
unnecessary string copies and avoiding throwing exceptions internally on
error.
Reviewed By: wez
Differential Revision: D15428700
fbshipit-source-id: feb73a1f53775fbf3d3acb8585fa689f79c0db81
Summary:
Add a thrift call to get the current config settings.
My primary use case for this method at the moment is to make it possible to
build integration tests that check the config behavior. However in the future
this will probably also be useful for building CLI commands to report the
current config values to allow debugging if there are ever issues. This API
can also be used to force EdenFS to immediately reload the config from disk.
Reviewed By: strager
Differential Revision: D15572124
fbshipit-source-id: da3bc982f9c419b3314a8b0560c9bd327760d429
Summary:
Change the `ReloadableConfig::getEdenConfig()` method to accept an enum to
control whether it should reload the configuration, rather than the simpler
bool that it used previously.
The primary motivation for this is to allow us to force a reload to occur in
unit tests and integration tests, even if this operation would otherwise be
within the normal reload throttling time limit.
Reviewed By: strager
Differential Revision: D15572122
fbshipit-source-id: 9c638a415a451f26e7f5923f544724594ed4e0f1
Summary: Previously functions were defined per string type but since SSO detection is now templatized so can path memory estimation.
Reviewed By: chadaustin
Differential Revision: D15723098
fbshipit-source-id: df8dcd359a0f4f704eba337ebf5e382d87ca3abe
Summary: Previously hard-coded values for how long a string has to be for SSO, now just check if data pointer is inside string stuct
Reviewed By: chadaustin
Differential Revision: D15720268
fbshipit-source-id: 988e45648e8b96332587f8d2f021642407d3dac3
Summary:
Update the EdenServer::unmountAll() code to make sure to keep a shared_ptr
reference to the EdenMount while the EdenMount::unmount() call is running.
Previously this could cause use-after-free errors due to the EdenMount
object being destroyed while the `unmount()` logic was still running. In
particular, as soon as we call `privhelper->fuseUnmount()` the
fuseCompletionFuture could complete in a different thread. The
fuseCompletionFuture's callback (normally the lambda inside
`EdenServer::mount()` which triggers the call to
`EdenServer::mountFinished()`) may have the last outstanding shared_ptr
refcount to the `EdenMount` object. When it finishes in this other thread
it will then destroy the EdenMount, possibly before the
`EdenMount::unmount()` logic has completed.
Reviewed By: strager
Differential Revision: D15688674
fbshipit-source-id: 47b8b4fe022d2bca112558f9ef32bcdd169fefb0
Summary: JournalStats is currently O(# of deltas), updating it to be O(1)
Reviewed By: chadaustin
Differential Revision: D15718255
fbshipit-source-id: 1fb3f0b76d736bfa22195231c21d5f8b742fa1f7
Summary: Setting up structure to get the stats from Eden's Journal to ODS
Reviewed By: chadaustin
Differential Revision: D15684380
fbshipit-source-id: 90debc3f2a09d497201e5e73f85400a994b7afba
Summary: Replace inclusions of `folly/futures/helpers.h` with `folly/futures/Future.h` to avoid the cyclic include trap.
Differential Revision: D15600549
fbshipit-source-id: 19950be24a7437fb1fbec293e24058adf17343ca
Summary:
The `testSplayOn()` and `testSplayOff()` tests have been failing occasionally
in continuous integration tests. In every test failure I have looked at, the
only reported failures were for iteration 0: the very first time the task was
scheduled after the server started. I believe the server start is simply
taking a non-zero amount of time, delaying how long it takes for the first
iteration of the tasks to be run.
This updates the tests to wait for 3 iterations of the main event base loop
before recording the test start time and scheduling the tasks.
Reviewed By: strager
Differential Revision: D15618328
fbshipit-source-id: 2b119700662457c0fcdf38cc231c6c7d26ade248
Summary:
Move the ConfigSource enum definition to a thrift file. This will let us
return ConfigSource values over thrift APIs in the future. This also allows
us to use thrift's `TEnumTraits` functionality to determine the maximum enum
value, rather than having to maintain a separate `kConfigSourceLastIndex`
variable.
As part of this change I also renamed the enum values to be CamelCase to match
our current C++ style recommendations and to avoid possibly conflicting with
macros defined in other headers (`DEFAULT` seemed particularly susceptible to
collision).
Reviewed By: strager
Differential Revision: D15572120
fbshipit-source-id: 8fbd03da221a9f75ef670dee1eb250eb198a5bd0
Summary:
Update the thrift definition for `initiateShutdown()` to indicate that it may
throw `EdenError` on failure. Without this all application-level errors that
occur will be translated to generic thrift `TApplicationException` objects on
the wire.
Also move this method declaration out of the "debugging APIs" section of the
file.
Reviewed By: strager
Differential Revision: D15572121
fbshipit-source-id: 7e621a24abd4347cedbb1bcce1ae9c2b70f991fd
Summary:
Don't honor the older (and deprecated) fb303 shutdown() call. The
`initiateShutdown()` call should always be used instead now, which includes a
bit more diagnostic information that we log to describe why the shutdown was
requested.
Reviewed By: wez, strager
Differential Revision: D15572123
fbshipit-source-id: 413be611313c9ae490673a2b34742e512cf8e5ea
Summary:
This was accidentally left behind from my original version of this code that
did not use `TEST_F()` but instead had a local `TestServer` variable in each
test function.
Reviewed By: wez
Differential Revision: D15618327
fbshipit-source-id: 5dd1252c77c286d22a53dd5cb4708a0b890f887c
Summary: This turned out to be a lot simpler than I thought it would!
Reviewed By: chadaustin
Differential Revision: D15630515
fbshipit-source-id: 51aeb8b6739cb886c3bca23ab441874ea9ac819c
Summary: These functions are part of the blocking API, which is being replaced by the Future-based API. They are not being called, so they are unneeded.
Reviewed By: chadaustin, strager
Differential Revision: D15582641
fbshipit-source-id: 11f693040124846515aef7c7ee6ad750e1b5c164
Summary: Developers should be encouraged to use Future-based functions instead of blocking functions in production code. Removing this blocking function makes it less convenient for developers to write blocking code in the future.
Reviewed By: strager
Differential Revision: D15564952
fbshipit-source-id: 3f62db472a59a6487ffe12e48c04178cad84ca61
Summary: The new Windows CMake targets were failing to include the generated thrift headers on Sandcastle machines. Making eden_service_thrift a dependecy will make sure that the thrift generated files are created before we compile the code.
Reviewed By: simpkins
Differential Revision: D15536743
fbshipit-source-id: 4bedb8f33cddf5f7eb8f5b5ce52b2013728b38f7
Summary: This is a stop gap solution to get the Windows build working with the getdeps, until we fix the SDK.
Reviewed By: chadaustin
Differential Revision: D15536740
fbshipit-source-id: 77cc6ea80c304a6cfcd0180bb28f63ce4dac2988
Summary:
Update EdenServer to use the new PeriodicTask class for flushing thread local
stats and collecting memory usage stats.
Reviewed By: strager
Differential Revision: D15492169
fbshipit-source-id: 41c23630ffe2f8365a0234a252821d07d48c8b29
Summary:
Add a new class for running periodic tasks on the main EventBase thread.
We already have several tasks that we run periodically:
- Flushing thread local stats (every 1s)
- Reporting memory stats (every 30s)
- Unloading unused inodes (configurable, but disabled by default)
I plan to add some new periodic tasks soon:
- Report LocalStore stats and perform garbage collection if needed
- Check if the config files have been updated and need to be re-read
This new class should make it easier to add periodic tasks with configurable
intervals, and make the intervals at which they run controllable from the
configuration file.
Reviewed By: strager
Differential Revision: D15492165
fbshipit-source-id: c1d2663da3c08851b60ba64d7a1e38ccca709b8a
Summary:
This diff teach Eden's Python client to use Windows' Unix Domain Socket support to connect to Eden.
#delayed_publish
Reviewed By: chadaustin
Differential Revision: D15454514
fbshipit-source-id: 86aa39f8aa13ae789efabda3fa88635f471a1a7a
Summary:
Some threads update FUSE counters, some threads update HgBackingStore counters, and some threads update HgImporter counters. No thread updates all of FUSE counters, HgBackingStore counters, and HgImporter counters. FUSE threads and HgImporter threads allocate the full set of counters (EdenThreadStats), even though many of the allocated counters can never be used during the thread's lifetime.
Reduce redundant allocation (and overhead during counter aggregation): split EdenThreadStats into three classes (FuseThreadStats, HgBackingStoreThreadStats, and HgImporterThreadStats) so different threads can allocate different sets of counters.
This diff should not change observable behavior.
Reviewed By: simpkins
Differential Revision: D14822273
fbshipit-source-id: cfd238187d20a0b8d3959673401ecad894e2095b
