Summary: Uses the existing RequestData class to make calls to static functions to set and get the `didImportFromBackingStore` flag.
Reviewed By: simpkins
Differential Revision: D16461868
fbshipit-source-id: e3ed39249f5dd1a842ad06a204b5933014b12f7f
Summary: hg debugedenimporthelper should either have the repo path as argument or the current working directory should be inside a repo. Setting the current working directory for the process.
Reviewed By: strager
Differential Revision: D16565042
fbshipit-source-id: d6e826bebf58fd58f5250e6c477849ca147628c8
Summary:
Update `EdenServer::mount()` to correctly handle errors that occur during the
mount `INITIALIZING` phase. Previously the code did not add error callbacks
to the `Future` result to handle errors during initialization. As a result we
would propagate the exception back to the thrift caller, but the `EdenMount`
object would remain in our mount point list, stuck forever in the
`INITIALIZING` state.
Reviewed By: strager
Differential Revision: D16590032
fbshipit-source-id: 9adbdf05441dad815096b195ece36f3d958c96a9
Summary:
Update `FsOverlay::validateHeader()` to throw `EdenError` exceptions rather
than `std::system_error`. These exceptions are generated when we find invalid
data, rather than from system calls that return errno values. Previously the
code was using `folly::throwSystemErrorExplicit()` and
`folly:throwSystemError()`, and passing in a made-up `EIO` value.
However, the latter two calls were incorrectly using
`folly::throwSystemError()` rather than `throwSystemErrorExplicit()`. As a
result the `EIO` parameter was being treated as part of the error message, and
the code was appending the error description for whatever value `errno`
happened to be set to at the moment.
This fixes the code to just throw `EdenError` exception types, since these
errors aren't caused by a real `errno` value.
Reviewed By: strager
Differential Revision: D16577698
fbshipit-source-id: 9487c34f04da99d397611f005f00f02114b12094
Summary:
Fix `FsOverlay::initOverlay()` to also create the `tmp` subdirectory, which is
required for the `FsOverlay` code to function properly. Previously this
directory was created by the `Overlay` class. However, `FsOverlay` should be
responsible for creating it so it can be used from other locations besides
just `Overlay`.
I also removed the code that checks if the `tmp` directory needs to be created
on restart. The code to create `tmp` has been present for over a year
(D8330070), and we no longer have anyone using EdenFS versions older than
this.
Reviewed By: strager
Differential Revision: D16577700
fbshipit-source-id: 3eaa28003159a03084c4f367c7b96d69fc1702d0
Summary:
There is no point to call `readBE<uint64_t>()` here. The only way these calls
can fail is if the buffer is too short for the data, but we already checked
the buffer length at the top of the function.
Reviewed By: strager
Differential Revision: D16577701
fbshipit-source-id: 8b57cec05453b9b7224470671e032e3aefe41033
Summary: Replace the uses of std::unique_ptr<JournalDelta> with just the JournalDelta itself to avoid an extra allocation.
Reviewed By: chadaustin, strager
Differential Revision: D16572089
fbshipit-source-id: be080b2fb9096f6c8783e2ecae21a99466336f6f
Summary:
Adds a debug command to eden such that users can flush the journal and cause the subscribers to get a truncated result, this will be useful for debugging [to debug we won't have to lower the memory limit and repeatedly touching files to fill up the journal].
Can be used with "eden debug flush_journal"
Reviewed By: chadaustin
Differential Revision: D16348811
fbshipit-source-id: fdbe6729d0393c424addcd42a091de440383035b
Summary: Refactoring the addDelta function to allow other Journal functions to add deltas while holding the deltaState lock. This refactoring will be used to create the flush function for the journal (which needs to hold the lock to empty the journal but wants to add a delta before releasing the lock).
Reviewed By: chadaustin
Differential Revision: D16341196
fbshipit-source-id: 7b7b1d933802b466efe624378206c72c71469129
Summary: Delete tests that rely on flatmanifest support in Mercurial.
Reviewed By: strager
Differential Revision: D15302489
fbshipit-source-id: 7e46f21bfe3ae910e205e73bca95d9a33f4a6816
Summary: D16461081 broke the windows build of Eden by not updating the Windows mount code, this diff makes the same change as that diff in the Windows portion of the code.
Reviewed By: chadaustin
Differential Revision: D16549392
fbshipit-source-id: eee1bede08c1b27329b026746dda6863519ed10b
Summary: Keep track of the longest query we had to resolve in accumulateRange. This stat will be helpful in determining how large the memory limit of the journal should be since we will know how far back people usually need to go.
Reviewed By: strager
Differential Revision: D16227920
fbshipit-source-id: a41c3b9f16b701cd8165e20409888983b8899dab
Summary: The journal will now keep of how many reads from accumulateRange have been truncated. This is a useful metric that will allow us to see how often we are forcing Watchman to use its fallback of creating a fresh instance and help us in calibrating if we find our memory limit is too small.
Reviewed By: strager
Differential Revision: D16017968
fbshipit-source-id: 95f4fbd1fd2d8523ff397202172408e1c89669be
Summary: D16096960 accidentally caused the truncation code to be run twice (once inline and once via a function code), this shouldn't cause any difference in outcome but is unnecessary.
Reviewed By: strager
Differential Revision: D16508798
fbshipit-source-id: 12781aee98e70e5105c5476d29cf5cdd1e31062d
Summary:
D15528156 inadvertently changed the return type of the `getPid()` call from
`i64` to `i32`. This is a backwards-incompatible change, and causes new
clients to reject the 64-bit response from older EdenFS daemons. This breaks
`edenfsctl status`, `edenfsctl clone` and other commands.
Reviewed By: pkaush
Differential Revision: D16522765
fbshipit-source-id: eecd344ee4b963d638576f146a87fc88a5003e55
Summary: Set up the infrastructure to add in Timeseries to the journal so that we can add in stats for the journal that relate to the whole process. For example, allow us to add in a truncatedRead TimeSeries easily as done in D16017968
Reviewed By: chadaustin
Differential Revision: D16461081
fbshipit-source-id: 964ff32e62aed0369da434793491b857c136b074
Summary: Now that Eden depends on open source fb303, EDEN_HAVE_STATS is unnecessary. Remove it.
Reviewed By: simpkins, strager
Differential Revision: D15526905
fbshipit-source-id: 2354f1b92545a089de0e91e7c33515fa0b74b067
Summary: Add a dependency from the eden open source build to the fb303 open source build and switch EdenServiceHandler to BaseService.
Reviewed By: simpkins
Differential Revision: D15528156
fbshipit-source-id: 2ca5c31dd9fcc9bac43fd399b27f33b6f2c5ebfc
Summary:
From andrewjcg:
> Basically, rocksdb headers exist at both e.g. rocksdb/src/include/rocksdb/flush_block_policy.h and rocksdb//flush_block_policy.h. The latter are never meant to be used, but Apple platforms by default add the repo root to the include dir before anything else. This means depending on file-relative inclusion from within rocksdb, we end up getting different versions of the header, which busts through #pragma once causing multiple definition errors.
This is split out from D15125826.
Reviewed By: andrewjcg
Differential Revision: D16353321
fbshipit-source-id: 095e3b7fb4f56b0be42d18fe4009336b65c8eb5e
Summary:
To save on memory the journal will now compact the same action repeated multiple times into the same action. This means that modifying the same file 100 times in a row results in 1 Journal delta instead of 100. [The results will cause Watchman to act the same since all queries are down from the current time, changes should only be visible by the number of deltas in the journal, how much memory the deltas are using, "eden debug journal" which will show that sequenceID's were skipped, and the fromSequence/fromTime returned by accumulateRange might be different]
**Memory Improvements:**
For buck commands, 1 run was conducted for each with a buck clean done before each build and then eden being restarted (so the clean did not affect the outcome) [results are formatted as 'with compaction' / 'without compaction']
“buck build mode/opt eden”
Entries: 154145 / 206108 [25.2% reduction]
Memory: 46.2 MB / 61.4 MB [24.7% reduction]
“buck build mode/opt warm_storage/common/...”
Entries: 318820 / 405016 [21.3% reduction]
Memory: 95.8 MB / 121.5 MB [21.2% reduction]
For Nuclide the result was calculated by getting the number of entries in the journal vs the last sequence ID in the journal ('entries we actually have' / 'entries we would have without compaction')
Using Nuclide’s Smart Log and Checking Out various commits / arc pulling:
Entries: 6091 / 23671 [74.3% reduction]
Reviewed By: chadaustin
Differential Revision: D16096960
fbshipit-source-id: f542ae32c889ebc9da442285d808ce75247f7e65
Summary:
This diff updates Eden's journal to be bounded in terms of memory usage which should help lessen the likelihood of Eden OOMing and taking up a large amount of our users' resources.
The memory limit is set to be 1 GB per journal [so a user with 3 mounts could expect the journals to possibly use up to 3 GB of memory].
The landing of this diff will need to wait until a version of Watchman that can handle truncation is deployed on all machines using Eden. This means we need to wait for a version of Watchman with D16219267 to fully land to machines.
Reviewed By: strager
Differential Revision: D15954994
fbshipit-source-id: 9a6425527f10a1ce051feb8fc7d092a84712f338
Summary:
We install the python files under `eden/fs/service` in a package named
`eden/thrift` in the built python binaries that use these modules. This
moves the source files into an `eden/thrift` subdirectory so that the source
directory layout more closely matches the final binary layout.
This will make it easier to run several of Eden's Python-based tools directly
from the source tree, without having to do as much directory layout
transformation. This is particularly helpful on platforms like Windows, which
don't currently have an equivalent of "live PARs" which can be run from the
source tree without requiring a rebuild after each file edit.
Reviewed By: chadaustin
Differential Revision: D16354622
fbshipit-source-id: 4b58cc96451b1ee5441714aaf74f5e3b6ada9eaa
Summary:
Let's minimize the amount of work we do before dropping privileges.
This shouldn't matter too much, since it is just a memory allocation and copy,
but I think we should keep the `dropPrivileges()` call as the very first thing
in `main()` to help prevent people from adding new functionaliy before it in
the future.
This also adds some block comments to help encourage people to only put code
after the "Root privileged dropped" comment moving forwards.
Reviewed By: chadaustin
Differential Revision: D16371783
fbshipit-source-id: 1ce5115bfa71565f0fe12ac9c9442bddb771ed48
Summary: In a previous diff we added the support to pass the command line to EdenServer but missed from the Windows main(). Until we unify the main function we need to add it at two places.
Reviewed By: simpkins
Differential Revision: D16370228
fbshipit-source-id: 47b9842fbca709e1d3c832db3d82765f2e5f8930
Summary:
I made the mistake of migrating the Thrift API for edenfs process
checks without a fallback for older edenfs running processes.
If getDaemonInfo() is not available, fall back on getPid().
Reviewed By: strager, pkaush
Differential Revision: D16365848
fbshipit-source-id: aee5c43c8c40db9a38fef7ca601f7639daa4adb8
Summary:
Open source fb303 will not have getPid() or getCommandLine(), so
introduce a new method for Eden's tests.
Reviewed By: fanzeyi
Differential Revision: D16292993
fbshipit-source-id: 5cdc006ec0ee15f50a3e1cebe9b46a3ea275ff78
Summary: I found out that the journal stats callbacks that were getting registered were not getting unregistered, this diff fixes that.
Reviewed By: chadaustin
Differential Revision: D16187569
fbshipit-source-id: 8c84e1515e376ccd7036a22c06e2e6b98dc62342
Summary: Updating Eden to send its blob cache memory usage to ODS. This will be useful in seeing how much of our overall memory usage is coming from the Blob Cache and debugging what is using large portions of Eden's memory.
Reviewed By: chadaustin
Differential Revision: D16157709
fbshipit-source-id: abae8e8da22369c7757187f998a1f6bef31d74e9
Summary: Running `eden debug journal -f` will print and follow the eden journal in a similar style to the unix `tail -f` command.
Reviewed By: chadaustin
Differential Revision: D16112458
fbshipit-source-id: 5304cd0f857bdbeca41c2591e98920f4f1fc8f42
Summary: Sets up a thrift interface to set the size of the journal (until truncation is added in the size field in the journal currently does nothing other than being viewable from getMemoryLimit)
Reviewed By: chadaustin
Differential Revision: D16042286
fbshipit-source-id: bc0acdf4ac5516cfac66fa0fbd87254d08ad479b
Summary: Added the cli command `eden stats object-store` for querying the counts on what part of the object store was responsible for finding the blob or blob size (local store or backing store). This will tell us how well local and in-memory caching works for different workflows.
Reviewed By: chadaustin
Differential Revision: D15934535
fbshipit-source-id: 70345f11a51c3c6996dc001d4101744395a3d182
Summary: eden_win_mount, eden_win_utils and eden_win_store include journal which depend on streamingeden_thrift. Adding the dependency to fix the build order.
Reviewed By: fanzeyi
Differential Revision: D16065869
fbshipit-source-id: d3bd5887e6c675885d284671bef7e514515d5fc0
Summary: Journal not specifying template argument for optional was causing eden builds to fail for old versions of C++. See D15978960.
Reviewed By: chadaustin
Differential Revision: D16059722
fbshipit-source-id: fb9903e304d9c5f79fc2d34f7763c3ada6ae4553
Summary: Create a counter for the end-to-end duration of the journal.
Reviewed By: chadaustin
Differential Revision: D15993675
fbshipit-source-id: 260a58a7b37dfa16b82b3df187ad7f7263ff67cf
Summary: Shows the end-to-end duration of the journal in "eden stats"
Reviewed By: chadaustin
Differential Revision: D15993261
fbshipit-source-id: 46471faca17d4f12ccdd8cea55b2722e33519a74
Summary:
Moving from the previous linked list approach to having a deque in the Journal.
The memory used by the deque is not tracked currently, the difference might be negligible though. I ran a script that touched a file 62 million times and here is the result comparing rss_memory used in eden and the journal's estimated memory: https://fburl.com/ods/6sq8g5vc and it still seems to closely estimate the right amount.
Reviewed By: strager
Differential Revision: D15944614
fbshipit-source-id: 6a1ac34ecd80c0eecb80411984f88f62ae712e91
Summary: Change getLatest such that it just returns Info about the latest delta and not the delta itself.
Reviewed By: strager
Differential Revision: D15931214
fbshipit-source-id: c7a1cf4d62cdd4f9396fab46354eabcbb31f32c0
Summary: Combine the backend function that iterates over the deltas into one function so we do not need to modify the code in multiple places as we start to modify the internal structure (Additionally pulled the one use of the code outside of the journal into the journal)
Reviewed By: strager
Differential Revision: D15912907
fbshipit-source-id: 2f59ff9e7f33ffa5b420859153a609e68bda10b4
Summary: Prior to this diff "eden debug journal" got the latest journal entry and manually seeked for the one "limit" prior to it, this has been updated to just passing the limit to the journal. [This will make the queries always happen from the tip of the journal, since in theory currently something can be added to the journal after the python script gets the latest delta]
Reviewed By: strager
Differential Revision: D15972018
fbshipit-source-id: 0ee0dd88a1e9edef5ccce3b3da2dbc09aa64f8a9
Summary:
Implements size-only local storage, as opposed to storing metadata. This is useful when the blob's SHA-1 is not needed. This diff prevents SHA-1 computations, which can be especially expensive for large blobs.
From D15934535, operations such as `ls -l` and `stat` will get the size of a blob in two ways:
1) The blob's size is already stored locally, so it will be deserialized and returned.
2) The blob is fetched from the backing store, stored, and its size is returned.
This diff optimizes the second case, because SHA-1 is no longer computed.
Reviewed By: strager
Differential Revision: D15723239
fbshipit-source-id: a868f3bf6b68a83ddafb057dc3e4e65f0a2dd989
Summary:
These allow the cli to setup and tear down mounts and
have the eden server keep track of them.
Reviewed By: strager
Differential Revision: D15707318
fbshipit-source-id: abdb8eaa28c8c67c8211a8af1647efe3a083e998
Summary: We need --edenDir support to run muliple instance of Edenfs, which is required to run the integration tests.
Reviewed By: simpkins
Differential Revision: D15951483
fbshipit-source-id: a516159cdeb5f046f795fc28399a2af5fe8a9f95
Summary: Adding a nullptr check for the case when the journal is empty and removing the use of default parameters.
Reviewed By: strager
Differential Revision: D15907329
fbshipit-source-id: 787b4a44f835fd8d128496ee6655e02987db98a7
Summary:
This contains several fixes to LocalStore handling during shutdown.
- Have EdenServer explicitly call localStore_->close() during shutdown.
This ensures that the local store really gets close, just in case some other
part of the code somehow still has an outstanding shared_ptr reference to
it.
- Add synchronization around internal access to the RocksDB object in
RocksDbLocalStore. This ensures that calling `close()` is safe even if
there happens to still be some outstanding I/O operations. In particular
this helps ensure that if background GC operation is in progress that
`close()` will wait until it completes before destroying the DB object.
This also improves the code so that calling subsequent methods on a closed
RocksDbLocalStore throws an exception, instead of simply crashing.
I don't believe the additional synchronization in RocksDbLocalStore should
have much impact on performance: the synchronization overhead should be very
low compared to the cost of the RocksDB reads/writes.
Ideally some of this synchronization logic should perhaps be moved into the
base `LocalStore` class: all of the different `LocalStore` implementations
should ideally ensure that `close()` is thread-safe and blocks until other
pending I/O operations are complete. However, that requires a bigger
refactoring. I may attempt that in a subsequent diff, but for now I mainly
want to address this problem just for RocksDbLocalStore.
Reviewed By: strager
Differential Revision: D15948382
fbshipit-source-id: 96d633ac0879b3321f596224907fcfe72691b3f0
Summary: This diff takes care of importing blob from Mononoke and Mercurial at the same time, also improves the name situation in the statistics counters.
Reviewed By: strager
Differential Revision: D15768557
fbshipit-source-id: 10cf831b1ae6dc9e6b91f1e96508c4fa92583743
Summary:
watchman/eden have been using the new Thrift streaming for nearly a year now and are the last users of the old
Thrift streaming. This diff kills the fallback `legacySubscribe` method, which complete's migration to the new Thrift
streaming and unblocks the complete removal of wangle-based streaming from Thrift and fbcode.
Reviewed By: wez
Differential Revision: D15878892
fbshipit-source-id: ec0d270dba79e56c7e41afbf36669a08e5a15518
Summary: This code calls close on fd while exiting. If the file doesn't exist the open will return -1 and close (-1) will be called.
Reviewed By: strager
Differential Revision: D15951485
fbshipit-source-id: ea3a52517847d75e9a822e51f360be7cb2c411da
