Summary:
To save on memory the journal will now compact the same action repeated multiple times into the same action. This means that modifying the same file 100 times in a row results in 1 Journal delta instead of 100. [The results will cause Watchman to act the same since all queries are down from the current time, changes should only be visible by the number of deltas in the journal, how much memory the deltas are using, "eden debug journal" which will show that sequenceID's were skipped, and the fromSequence/fromTime returned by accumulateRange might be different]
**Memory Improvements:**
For buck commands, 1 run was conducted for each with a buck clean done before each build and then eden being restarted (so the clean did not affect the outcome) [results are formatted as 'with compaction' / 'without compaction']
“buck build mode/opt eden”
Entries: 154145 / 206108 [25.2% reduction]
Memory: 46.2 MB / 61.4 MB [24.7% reduction]
“buck build mode/opt warm_storage/common/...”
Entries: 318820 / 405016 [21.3% reduction]
Memory: 95.8 MB / 121.5 MB [21.2% reduction]
For Nuclide the result was calculated by getting the number of entries in the journal vs the last sequence ID in the journal ('entries we actually have' / 'entries we would have without compaction')
Using Nuclide’s Smart Log and Checking Out various commits / arc pulling:
Entries: 6091 / 23671 [74.3% reduction]
Reviewed By: chadaustin
Differential Revision: D16096960
fbshipit-source-id: f542ae32c889ebc9da442285d808ce75247f7e65
Summary:
This diff updates Eden's journal to be bounded in terms of memory usage which should help lessen the likelihood of Eden OOMing and taking up a large amount of our users' resources.
The memory limit is set to be 1 GB per journal [so a user with 3 mounts could expect the journals to possibly use up to 3 GB of memory].
The landing of this diff will need to wait until a version of Watchman that can handle truncation is deployed on all machines using Eden. This means we need to wait for a version of Watchman with D16219267 to fully land to machines.
Reviewed By: strager
Differential Revision: D15954994
fbshipit-source-id: 9a6425527f10a1ce051feb8fc7d092a84712f338
Summary: Running `eden debug journal -f` will print and follow the eden journal in a similar style to the unix `tail -f` command.
Reviewed By: chadaustin
Differential Revision: D16112458
fbshipit-source-id: 5304cd0f857bdbeca41c2591e98920f4f1fc8f42
Summary: Sets up a thrift interface to set the size of the journal (until truncation is added in the size field in the journal currently does nothing other than being viewable from getMemoryLimit)
Reviewed By: chadaustin
Differential Revision: D16042286
fbshipit-source-id: bc0acdf4ac5516cfac66fa0fbd87254d08ad479b
Summary: Journal not specifying template argument for optional was causing eden builds to fail for old versions of C++. See D15978960.
Reviewed By: chadaustin
Differential Revision: D16059722
fbshipit-source-id: fb9903e304d9c5f79fc2d34f7763c3ada6ae4553
Summary:
Moving from the previous linked list approach to having a deque in the Journal.
The memory used by the deque is not tracked currently, the difference might be negligible though. I ran a script that touched a file 62 million times and here is the result comparing rss_memory used in eden and the journal's estimated memory: https://fburl.com/ods/6sq8g5vc and it still seems to closely estimate the right amount.
Reviewed By: strager
Differential Revision: D15944614
fbshipit-source-id: 6a1ac34ecd80c0eecb80411984f88f62ae712e91
Summary: Change getLatest such that it just returns Info about the latest delta and not the delta itself.
Reviewed By: strager
Differential Revision: D15931214
fbshipit-source-id: c7a1cf4d62cdd4f9396fab46354eabcbb31f32c0
Summary: Combine the backend function that iterates over the deltas into one function so we do not need to modify the code in multiple places as we start to modify the internal structure (Additionally pulled the one use of the code outside of the journal into the journal)
Reviewed By: strager
Differential Revision: D15912907
fbshipit-source-id: 2f59ff9e7f33ffa5b420859153a609e68bda10b4
Summary: Adding a nullptr check for the case when the journal is empty and removing the use of default parameters.
Reviewed By: strager
Differential Revision: D15907329
fbshipit-source-id: 787b4a44f835fd8d128496ee6655e02987db98a7
Summary:
Update the copyright & license headers in C++ files to reflect the
relicensing to GPLv2
Reviewed By: wez
Differential Revision: D15487078
fbshipit-source-id: 19f24c933a64ecad0d3a692d0f8d2a38b4194b1d
Summary: Merge was a function on a JournalDelta that created a new JournalDelta and optionally left it connected to old JournalDeltas. AccumulateRange is a new function on the Journal itself (acting on the latest delta) that creates a JournalDeltaSum (which can't be connected to older Deltas)
Reviewed By: chadaustin
Differential Revision: D15881452
fbshipit-source-id: 573505c1171f78d46afc98f1db9b5b9ee2fff60f
Summary: Making addDelta private and giving users a more user-friendly way of appending entries to the journal.
Reviewed By: chadaustin, strager
Differential Revision: D15868089
fbshipit-source-id: 00c8a3066f0e4483e3c792651ade5f6a7ea05eed
Summary: JournalStats is currently O(# of deltas), updating it to be O(1)
Reviewed By: chadaustin
Differential Revision: D15718255
fbshipit-source-id: 1fb3f0b76d736bfa22195231c21d5f8b742fa1f7
Summary:
When running the watchman integration tests, I saw that some of them were
fataling here:
diffusion/FBS/browse/master/fbcode/thrift/lib/cpp2/async/StreamPublisher-inl.h$71
In this situation we're tearing down all the subscribers via
`Journal::cancelAllSubscribers` and this causes the stream be destroyed and
trigger this FATAL check.
We need to explicitly complete it before we can proceeed, so we do the gross
thing and call this in the destructor of a little wrapper.
This is made a little more complex because the disconnection handler has to
inform the Journal that it was torn down in the case where the client has
gone away. The result is that we need to be careful to avoid the destructor
on Journal callback from running while we hold the Journal subscriber lock.
Reviewed By: strager
Differential Revision: D10024271
fbshipit-source-id: 06a9d40f7f6e46fe35ffcedba2669e27e6624427
Summary:
For O(1) stack space JournalDelta destruction, we need to check
whether a reference-counted pointer has a unique refcount. It looks at
first glance like shared_ptr<T>::unique() gives us that, but it uses a
relaxed load and is now deprecated.
Sadly, the easiest thing is to make a new smart pointer class. (It
happens to be slightly more efficient, requiring one fewer heap
allocation per entry in the chain.)
Reviewed By: simpkins
Differential Revision: D9355314
fbshipit-source-id: 8c782ba9e0ec27fae90325079c199e1b82df88fa
Summary:
This is follow-up to the lock ordering issues in
StreamingSubscriber. The Journal locks are now finer-grained and no
locks are held while the subscribers are invoked. This change
prevents future deadlocks.
Reviewed By: wez
Differential Revision: D6281410
fbshipit-source-id: 797c164395831752f61cc15928b6d4ce4dab1b68
Summary:
Per discussion with bolinfest, this brings Eden in line with clang-format.
This diff was generated with `find . \( -iname '*.cpp' -o -iname '*.h' \) -exec bash -c "yes | arc lint {}" \;`
Reviewed By: bolinfest
Differential Revision: D6232695
fbshipit-source-id: d54942bf1c69b5b0dcd4df629f1f2d5538c9e28c
Summary:
We have a couple of issues with the watchman/eden integration:
1. If you "win" the race, you can cause a segfault during shutdown
by changing files during unmount. This causes the journal updating
code to trigger a send to the client, but the associated eventBase
has already been destroyed.
2. We don't proactively send any signal to the subscriber (in practice: watchman)
when we unmount. Watchman detects the eden shutdown by noticing that its
socket has closed but has no way to detect an unmount.
This diff tries to connect the unmount with the set of subscribers and tries
to cause the thrift socket to close out.
Reviewed By: bolinfest
Differential Revision: D6162717
fbshipit-source-id: 42d4a005089cd9cddf204997b1643570488f04c3
Summary:
This tweaks the definition of the subscribe method in the
streamingeden thrift file and implements the server side of the
thrift service, and the client side to consume it in watchman.
The definition is now a bit simpler than it was previously; we're
now just sending a stream of the updated JournalPosition objects
to the client rather than computing and sending FileDelta objects.
This is cheaper for Eden to deal with because it is very cheap to take
the current journal position and pass that over the wire. This is
important because a burst of mutations will likely queue up a bunch
of these in quick succession.
In a future diff I'm considering adding a latency parameter for
the subscription so that we can constrain the number of updates
over a certain time period (likely in the 10's of milliseconds range).
For now I just want to prove that the concept works.
On the watchman side we just need to pull these off the wire as they are sent
by eden, then wait for the subscription stream to settle before internally
broadcasting to any connected subscribers.
Reviewed By: bolinfest
Differential Revision: D4647259
fbshipit-source-id: 03aa16e59a43195a2505a8d03bce1ccf88a8d42f
Summary:
Update copyright statements to "2016-present". This makes our updated lint
rules happy and complies with the recommended license header statement.
Reviewed By: wez, bolinfest
Differential Revision: D4433594
fbshipit-source-id: e9ecb1c1fc66e4ec49c1f046c6a98d425b13bc27
Summary:
This just populates the initial snapshot hash in the journal.
The `addDelta` method will propagate this into subsequent deltas if the delta
to be added has hash values that have not been set from the default 0-filled
hash values.
Reviewed By: simpkins
Differential Revision: D3872936
fbshipit-source-id: d0014ded40488a2be04d5a381e1d9815c7f0a638
Summary:
This implements a pretty simple change Journal and associated
JournalDelta.
The Journal is intended to be held in memory and not persisted to disk.
The idea is that we'll hold a `Synchronized<Journal>` along with the
other mount data and grab a `wlock` on it each time we want to add
a change record.
This diff doesn't change any other existing functionality.
Reviewed By: simpkins
Differential Revision: D3660162
fbshipit-source-id: a6b6fa28dd12e4d34718956167ee87f8cb2d89ca
