Summary: Tracing was not an accurate name for what this directory had become. So rename it to telemetry.
Reviewed By: wez
Differential Revision: D17923303
fbshipit-source-id: fca07e8447d9b9b3ea5d860809a2d377e3c4f9f2
Summary:
Update the CMakeLists.txt to also build the Python-based `edenfsctl` command
line tool.
This requires switching most of the thrift rules to generate both C++ and
Python sources.
Note that one missing feature at this point is that this does not package
external dependencies into the binary. Currently `edenfsctl` depends on both
`six` and `toml` as external dependencies. For now these must be available in
your `PYTHONPATH` in order to run the generated `edenfsctl` binary.
Reviewed By: chadaustin
Differential Revision: D17127615
fbshipit-source-id: fc138ab39e75c6a5bbd39e3f527d4e9f7f420e46
Summary:
This fixes a few issues with the library dependencies:
- The `eden_utils` library depends on `eden_service_thrift`, not
`eden_service`. By incorrectly depending on `eden_service` this introduced
a circular dependency which would cause a build failure, depending on which
order CMake chose to try and emit the link line.
- The `eden_config` library depends on code from `eden_model` (for `Hash` and
`ParentCommits`)
- The `eden_inodes` library depends on `eden_model_git` for the `GitIgnore`
logic. I also alphabetized the dependency list.
Reviewed By: wez
Differential Revision: D17124930
fbshipit-source-id: 70cbe81081fc1dc807cca13a93edc25ba270b01f
Summary:
The toHash field is unnecessary in HashUpdateDeltas since we only ever iterate in reverse. We instead keep track of the current hash in the journal itself instead of looking at the toHash of the latest HashUpdateDelta.
Delta Struct Sizes:
> File Change: 88
> Hash Update: 96
Reviewed By: strager
Differential Revision: D16522519
fbshipit-source-id: 43baccc8ef2579f72609cc84e81e218794b11725
Summary: Create the second delta type HashUpdateJournalDelta that keeps track of changes to the hash [and the unclean files associated with that hash change]. Journal methods were updated to account for the different delta types.
Reviewed By: strager
Differential Revision: D16520444
fbshipit-source-id: 2a5cea11c9e70e30f6db55d9c8e33f9322ae91fc
Summary: Add in two tests for deltas that change commit hashes since a test for this did not exist before. Previously a bug in setting the fromHash of a result only showed up as a failure in Watchman's integration tests with Eden.
Reviewed By: chadaustin
Differential Revision: D16528248
fbshipit-source-id: 56eede749ef2da4dc492a1f7376dc07ca8aa3050
Summary: Deltas are stores in a deque and therefore not malloced independently so therefore there is no reason to be getting the goodMallocSize of them. We also need to do accounting for the buckets in the deque that the deltas fall into.
Reviewed By: strager
Differential Revision: D16566675
fbshipit-source-id: 4506fbbcc2044b8fdfe6244313ef7480cfa5151e
Summary: Replace the uses of std::unique_ptr<JournalDelta> with just the JournalDelta itself to avoid an extra allocation.
Reviewed By: chadaustin, strager
Differential Revision: D16572089
fbshipit-source-id: be080b2fb9096f6c8783e2ecae21a99466336f6f
Summary:
Adds a debug command to eden such that users can flush the journal and cause the subscribers to get a truncated result, this will be useful for debugging [to debug we won't have to lower the memory limit and repeatedly touching files to fill up the journal].
Can be used with "eden debug flush_journal"
Reviewed By: chadaustin
Differential Revision: D16348811
fbshipit-source-id: fdbe6729d0393c424addcd42a091de440383035b
Summary: Refactoring the addDelta function to allow other Journal functions to add deltas while holding the deltaState lock. This refactoring will be used to create the flush function for the journal (which needs to hold the lock to empty the journal but wants to add a delta before releasing the lock).
Reviewed By: chadaustin
Differential Revision: D16341196
fbshipit-source-id: 7b7b1d933802b466efe624378206c72c71469129
Summary: Keep track of the longest query we had to resolve in accumulateRange. This stat will be helpful in determining how large the memory limit of the journal should be since we will know how far back people usually need to go.
Reviewed By: strager
Differential Revision: D16227920
fbshipit-source-id: a41c3b9f16b701cd8165e20409888983b8899dab
Summary: The journal will now keep of how many reads from accumulateRange have been truncated. This is a useful metric that will allow us to see how often we are forcing Watchman to use its fallback of creating a fresh instance and help us in calibrating if we find our memory limit is too small.
Reviewed By: strager
Differential Revision: D16017968
fbshipit-source-id: 95f4fbd1fd2d8523ff397202172408e1c89669be
Summary: D16096960 accidentally caused the truncation code to be run twice (once inline and once via a function code), this shouldn't cause any difference in outcome but is unnecessary.
Reviewed By: strager
Differential Revision: D16508798
fbshipit-source-id: 12781aee98e70e5105c5476d29cf5cdd1e31062d
Summary: Set up the infrastructure to add in Timeseries to the journal so that we can add in stats for the journal that relate to the whole process. For example, allow us to add in a truncatedRead TimeSeries easily as done in D16017968
Reviewed By: chadaustin
Differential Revision: D16461081
fbshipit-source-id: 964ff32e62aed0369da434793491b857c136b074
Summary:
To save on memory the journal will now compact the same action repeated multiple times into the same action. This means that modifying the same file 100 times in a row results in 1 Journal delta instead of 100. [The results will cause Watchman to act the same since all queries are down from the current time, changes should only be visible by the number of deltas in the journal, how much memory the deltas are using, "eden debug journal" which will show that sequenceID's were skipped, and the fromSequence/fromTime returned by accumulateRange might be different]
**Memory Improvements:**
For buck commands, 1 run was conducted for each with a buck clean done before each build and then eden being restarted (so the clean did not affect the outcome) [results are formatted as 'with compaction' / 'without compaction']
“buck build mode/opt eden”
Entries: 154145 / 206108 [25.2% reduction]
Memory: 46.2 MB / 61.4 MB [24.7% reduction]
“buck build mode/opt warm_storage/common/...”
Entries: 318820 / 405016 [21.3% reduction]
Memory: 95.8 MB / 121.5 MB [21.2% reduction]
For Nuclide the result was calculated by getting the number of entries in the journal vs the last sequence ID in the journal ('entries we actually have' / 'entries we would have without compaction')
Using Nuclide’s Smart Log and Checking Out various commits / arc pulling:
Entries: 6091 / 23671 [74.3% reduction]
Reviewed By: chadaustin
Differential Revision: D16096960
fbshipit-source-id: f542ae32c889ebc9da442285d808ce75247f7e65
Summary:
This diff updates Eden's journal to be bounded in terms of memory usage which should help lessen the likelihood of Eden OOMing and taking up a large amount of our users' resources.
The memory limit is set to be 1 GB per journal [so a user with 3 mounts could expect the journals to possibly use up to 3 GB of memory].
The landing of this diff will need to wait until a version of Watchman that can handle truncation is deployed on all machines using Eden. This means we need to wait for a version of Watchman with D16219267 to fully land to machines.
Reviewed By: strager
Differential Revision: D15954994
fbshipit-source-id: 9a6425527f10a1ce051feb8fc7d092a84712f338
Summary: Running `eden debug journal -f` will print and follow the eden journal in a similar style to the unix `tail -f` command.
Reviewed By: chadaustin
Differential Revision: D16112458
fbshipit-source-id: 5304cd0f857bdbeca41c2591e98920f4f1fc8f42
Summary: Sets up a thrift interface to set the size of the journal (until truncation is added in the size field in the journal currently does nothing other than being viewable from getMemoryLimit)
Reviewed By: chadaustin
Differential Revision: D16042286
fbshipit-source-id: bc0acdf4ac5516cfac66fa0fbd87254d08ad479b
Summary: Journal not specifying template argument for optional was causing eden builds to fail for old versions of C++. See D15978960.
Reviewed By: chadaustin
Differential Revision: D16059722
fbshipit-source-id: fb9903e304d9c5f79fc2d34f7763c3ada6ae4553
Summary: Shows the end-to-end duration of the journal in "eden stats"
Reviewed By: chadaustin
Differential Revision: D15993261
fbshipit-source-id: 46471faca17d4f12ccdd8cea55b2722e33519a74
Summary:
Moving from the previous linked list approach to having a deque in the Journal.
The memory used by the deque is not tracked currently, the difference might be negligible though. I ran a script that touched a file 62 million times and here is the result comparing rss_memory used in eden and the journal's estimated memory: https://fburl.com/ods/6sq8g5vc and it still seems to closely estimate the right amount.
Reviewed By: strager
Differential Revision: D15944614
fbshipit-source-id: 6a1ac34ecd80c0eecb80411984f88f62ae712e91
Summary: Change getLatest such that it just returns Info about the latest delta and not the delta itself.
Reviewed By: strager
Differential Revision: D15931214
fbshipit-source-id: c7a1cf4d62cdd4f9396fab46354eabcbb31f32c0
Summary: Combine the backend function that iterates over the deltas into one function so we do not need to modify the code in multiple places as we start to modify the internal structure (Additionally pulled the one use of the code outside of the journal into the journal)
Reviewed By: strager
Differential Revision: D15912907
fbshipit-source-id: 2f59ff9e7f33ffa5b420859153a609e68bda10b4
Summary: Adding a nullptr check for the case when the journal is empty and removing the use of default parameters.
Reviewed By: strager
Differential Revision: D15907329
fbshipit-source-id: 787b4a44f835fd8d128496ee6655e02987db98a7
Summary:
Update the copyright & license headers in CMake files to reflect the
relicensing to GPLv2
Reviewed By: wez
Differential Revision: D15487079
fbshipit-source-id: 715e559464c19a0070d6e55a095b3fc7d61ad2f8
Summary:
Update the copyright & license headers in C++ files to reflect the
relicensing to GPLv2
Reviewed By: wez
Differential Revision: D15487078
fbshipit-source-id: 19f24c933a64ecad0d3a692d0f8d2a38b4194b1d
Summary: Merge was a function on a JournalDelta that created a new JournalDelta and optionally left it connected to old JournalDeltas. AccumulateRange is a new function on the Journal itself (acting on the latest delta) that creates a JournalDeltaSum (which can't be connected to older Deltas)
Reviewed By: chadaustin
Differential Revision: D15881452
fbshipit-source-id: 573505c1171f78d46afc98f1db9b5b9ee2fff60f
Summary: Making addDelta private and giving users a more user-friendly way of appending entries to the journal.
Reviewed By: chadaustin, strager
Differential Revision: D15868089
fbshipit-source-id: 00c8a3066f0e4483e3c792651ade5f6a7ea05eed
Summary:
Some newer versions of `clang` (such as Apple's version 11) will warn/error out if a constructor or assignment operator
marked `default` is implicitly deleted (e.g., if the object contains a non-moveable/non-copyable member). This diff
removes all such defaulted constructors/assignment operators, which I ran into while building `edenfs` on my Macbook Pro.
Reviewed By: chadaustin, strager
Differential Revision: D15901794
fbshipit-source-id: 794ed8377693a6735bb567635dc919bc678751a4
Summary: JournalStats is currently O(# of deltas), updating it to be O(1)
Reviewed By: chadaustin
Differential Revision: D15718255
fbshipit-source-id: 1fb3f0b76d736bfa22195231c21d5f8b742fa1f7
Summary:
When running the watchman integration tests, I saw that some of them were
fataling here:
diffusion/FBS/browse/master/fbcode/thrift/lib/cpp2/async/StreamPublisher-inl.h$71
In this situation we're tearing down all the subscribers via
`Journal::cancelAllSubscribers` and this causes the stream be destroyed and
trigger this FATAL check.
We need to explicitly complete it before we can proceeed, so we do the gross
thing and call this in the destructor of a little wrapper.
This is made a little more complex because the disconnection handler has to
inform the Journal that it was torn down in the case where the client has
gone away. The result is that we need to be careful to avoid the destructor
on Journal callback from running while we hold the Journal subscriber lock.
Reviewed By: strager
Differential Revision: D10024271
fbshipit-source-id: 06a9d40f7f6e46fe35ffcedba2669e27e6624427
Summary:
While destroying JournalDelta entries, unzip the chain as we go and
destroy each entry one by one. This prevents the stack from
overflowing on long JournalDelta chains and should fix some flaky
tests.
Reviewed By: wez
Differential Revision: D9355365
fbshipit-source-id: 31af124d318ca5d7a84314b707e1b3c71b2ccaa9
Summary:
For O(1) stack space JournalDelta destruction, we need to check
whether a reference-counted pointer has a unique refcount. It looks at
first glance like shared_ptr<T>::unique() gives us that, but it uses a
relaxed load and is now deprecated.
Sadly, the easiest thing is to make a new smart pointer class. (It
happens to be slightly more efficient, requiring one fewer heap
allocation per entry in the chain.)
Reviewed By: simpkins
Differential Revision: D9355314
fbshipit-source-id: 8c782ba9e0ec27fae90325079c199e1b82df88fa
Summary:
Watchman's Eden integration has a bug where the combination of
Watchman querying Eden for overlapping delta ranges ("give me changes
between X and Y, now changes between X+1 and Y+1") and Eden eliding
redundant change events ("add-modify-remove" -> []) results in
Watchman sometimes reporting that a file exists in its final
subscription update when it no longer does.
The fix is to never elide events, even for files that were added and
removed in the same sequence. To continue to support Watchman's `new`
flag, track whether a file existed at the beginning and end of a
journal delta.
Reviewed By: wez
Differential Revision: D9304964
fbshipit-source-id: f34c12b25f2b24e3a0d46fc94aa428528f4c5098
Summary:
Historically, we have seen a number of messages like the following in the Eden
logs:
```
Journal for .hg/blackbox.log holds invalid Created, Created sequence
```
Apparently we were getting these invalid sequences because we were not always
recording a "rename" correctly. The "rename" constructor for a `JournalDelta`
assumed that the source path should be included in the list of "removed" files
while the destination path should be included in the list of "created" files.
However, that is not accurate if the destination path already existed before
the user ran `mv`.
Fortunately, we already check whether the destination file exists in
`TreeInode::doRename()`, so it is straightforward to determine whether the
action is a "rename" (destination does not exist) or an "replace" (destination
already exists) and then classify the destination path accordingly.
As demonstrated by the new test introduced in this commit
(`JournalUpdateTest::moveFileReplace`), in the old implementation,
a file that was removed after it was overwritten would not show up as
removed in the merged `JournalDelta`. Because Watchman relies on
`JournalDelta::merge()` via the Thrift method `getFilesChangedSince()`,
this would cause Watchman to report such a file as still existing even
though it was removed.
This definitely caused bugs in Nuclide. It is likely that other tools that rely
on Watchman in Eden (such as Buck) may have also done incorrect things
because of this bug, so this could explain past reported issues.
Reviewed By: simpkins
Differential Revision: D7888249
fbshipit-source-id: 3e57963f27c5421a6175d1a759db8d9597ed76f3
Summary:
Promote the folly logging code out of the experimental subdirectory.
We have been using this for several months in a few projects and are pretty
happy with it so far.
After moving it out of the experimental/ subdirectory I plan to update
folly::Init() to automatically support configuring it via a `--logging` command
line flag (similar to the initialization it already does today for glog).
Reviewed By: yfeldblum, chadaustin
Differential Revision: D7755455
fbshipit-source-id: 052db34c97f7516728f7cbb1a5ad959def2f6efb
Summary:
This removes the TARGETS files from the eden github repository. The
open source buck build has been failing for several months, since buck
removed support for the thrift_library() rule.
I will potentially take a stab at adding CMake build support for Eden
at some point in the future.
Reviewed By: chadaustin
Differential Revision: D6893233
fbshipit-source-id: e6023094a807cf481ac49998c6f21b213be6c288
Summary:
This is a codemod to change from using @/ to // in basic cases.
- TARGETS files with lines starting with @/ (but excluding @/third-party:
- autodeps lines in source and TARGETS files ( (dep|manual)=@/ ), excluding @/third-party
- Targets in string macros
The only thing left of the old format should be @/third-party:foo:bar
drop-conflicts
Reviewed By: ttsugriy
Differential Revision: D6605465
fbshipit-source-id: ae50de2e1edb3f97c0b839d4021f38d77b7ab64c
Summary:
This is follow-up to the lock ordering issues in
StreamingSubscriber. The Journal locks are now finer-grained and no
locks are held while the subscribers are invoked. This change
prevents future deadlocks.
Reviewed By: wez
Differential Revision: D6281410
fbshipit-source-id: 797c164395831752f61cc15928b6d4ce4dab1b68
Summary:
Per discussion with bolinfest, this brings Eden in line with clang-format.
This diff was generated with `find . \( -iname '*.cpp' -o -iname '*.h' \) -exec bash -c "yes | arc lint {}" \;`
Reviewed By: bolinfest
Differential Revision: D6232695
fbshipit-source-id: d54942bf1c69b5b0dcd4df629f1f2d5538c9e28c
Summary:
We have a couple of issues with the watchman/eden integration:
1. If you "win" the race, you can cause a segfault during shutdown
by changing files during unmount. This causes the journal updating
code to trigger a send to the client, but the associated eventBase
has already been destroyed.
2. We don't proactively send any signal to the subscriber (in practice: watchman)
when we unmount. Watchman detects the eden shutdown by noticing that its
socket has closed but has no way to detect an unmount.
This diff tries to connect the unmount with the set of subscribers and tries
to cause the thrift socket to close out.
Reviewed By: bolinfest
Differential Revision: D6162717
fbshipit-source-id: 42d4a005089cd9cddf204997b1643570488f04c3
