Summary:
In the future, we will likely coalesce redundant fetches at the
BlobAccess/ObjectStore layer. To measure what we actually want,
populate a normal ObjectStore with a NullLocalStore and add counters
to FakeBackingStore.
Reviewed By: wez
Differential Revision: D13454331
fbshipit-source-id: 2fbf393d159f94e84c24ac53ccc207162fa754b7
Summary:
There was a bug in BlobCache where, if you had an interest handle to a
blob, but that blob was evicted anyway and then something else caused
it to be reloaded, dropping your interest handle would cause the blob
to be incorrectly evicted since the reference counts were no longer
compatible. Add a version to cache items and only decrement the
reference count on an item if the interest handle and item agree.
Reviewed By: strager
Differential Revision: D13405144
fbshipit-source-id: aee052bf777e7225551c3ae2b8b69a99f4f77691
Summary:
Have `eden stats` print the size of the blob cache if the running
edenfs has information about it.
Reviewed By: strager
Differential Revision: D13349220
fbshipit-source-id: 9f59f4399f2d4283aa80bcb54ba73c51d555d502
Summary:
A later diff needed a constant for the SHA-1 of an empty buffer. While
I'm at it, I made Hash a little bit nicer to use.
Reviewed By: strager
Differential Revision: D13224195
fbshipit-source-id: b2fb1437be042215b5b398a8c7fc9fc5dd115e9e
Summary: In support of making Eden's file access stateless, add a facade that connects the BlobCache and ObjectStore, allowing FileInode to fetch blobs, minimizing reloads and memory usage.
Reviewed By: strager
Differential Revision: D10850143
fbshipit-source-id: 4307f7c1143aecad1284ea3cadf3e4efca9a3925
Summary: Add a BlobCache with a maximum cache size and a minimum entry count and interest-based eviction.
Reviewed By: strager
Differential Revision: D12972062
fbshipit-source-id: 1958f7f500c051a5bc0b39b5b89a6f0fc1774b0f
Summary:
Update all of the C++ unit tests that create temporary files and directories
to use the new `facebook::eden::makeTempFile()` and
`facebook::eden::makeTempDir()` functions.
Note that this likely changes the behavior of some code paths in meaningful
ways: `/dev/shm` normally does not support `getxattr()`, and Eden's overlay
code attempts to store the SHA-1 for materialized files as using extended
attributes. This means that the tests will now typically hit the fallback
code path and will not store SHA-1 data in the overlay.
Reviewed By: chadaustin, strager
Differential Revision: D12971162
fbshipit-source-id: 6cc5eba2e04be7e9a13a30e90883feadfb78f9ce
Summary:
Because Mercurial blob IDs change without the contents changing, and
because files get unloaded upon checkout, rebasing across a large
distance in history can result in status fetching a lot of
metadata. Keep a smallish LRU cache for SHA-1 and size by blob ID.
Reviewed By: strager
Differential Revision: D10419965
fbshipit-source-id: 81499573814775471913db05f924767c3bab300e
Summary:
This is part of "the great r-valuification of folly::Future":
* This is something we should do for safety in general.
* Context: `Future::get(...)` means both `Future::get()` and `Future::get(Duration)`
* Using lvalue-qualified `Future::get(...)` has caused some failures around D7840699 since lvalue-qualification hides that operation's move-out semantics - leads to some use of future operations that are really not correct, but are not obviously incorrect.
* Problems with `Future::get(...) &`: it moves-out the result but doesn't invalidate the Future - the Future remains (technically) valid even though it actually is partially moved-out. Callers can subsequently access that moved-out result via things like `future.get(...)`, `future.result()`, `future.value()`, etc. - these access an already-moved-out result which is/can be surprising.
* Reasons `Future::get(...) &&` is better: its semantics are more obvious and user-testable. It moves-out the Future, leaving it with `future.valid() == false`.
Reviewed By: LeeHowes
Differential Revision: D8756764
fbshipit-source-id: 5c75c79cebcec77658a3a4605087716e969a376d
Summary:
Have getBlobMetadata always return a Future. It's a little unfortunate
that this will always allocate, but it sounds like we might decide to
put all RocksDB access on a background thread to increase CPU
parallelism.
Reviewed By: bolinfest
Differential Revision: D8101464
fbshipit-source-id: 6e9ec95050c366c7c57519e3f68b311470b2addd
Summary:
Remove getTreeFuture and have getTree always return a Future. It's a
little unfortunate that this will always allocate, but it sounds like
we might decide to put all RocksDB access on a background thread to
increase CPU parallelism.
Reviewed By: bolinfest
Differential Revision: D8101430
fbshipit-source-id: e12b7ab07b3468114a58753768655c107265b8af
Summary:
Remove getBlobFuture and have getBlob always return a Future. It's a
little unfortunate that this will always allocate, but it sounds like
we might decide to put all RocksDB access on a background thread to
increase CPU parallelism.
Reviewed By: bolinfest
Differential Revision: D8101402
fbshipit-source-id: d6cbbd7fe4fe55bad661c9158297db2f03f7d352
Summary:
Add a clearCaches function to LocalStore that deletes all data from
LocalStore that could be retrieved from Mercurial.
Reviewed By: wez
Differential Revision: D8101365
fbshipit-source-id: d46d0db94e6f85aaf542d9f6b9b96fbdcc548b57
Summary:
This is not at all clear from cppreference.com, but per
https://www.youtube.com/watch?v=dTeKf5Oek2c, it sounds to me like
recommended practice is to either:
`using namespace std::chrono_literals` (or string_literals or
whatever) to pull in a focused set of literals.
or
`using namespace std::literals` to pull in all standard literals.
or
`using namespace std` to pull in everything.
`using namespace std::literals::chrono_literals` is unnecessarily
verbose.
Adopt those standards in Eden.
Reviewed By: simpkins
Differential Revision: D8060944
fbshipit-source-id: 4d9dd4329698b7ff5e5c81b5b28780ca4d81a2a1
Summary:
I got tired of typing PathComponentPiece{"..."} in tests so here are
some operator literals.
Reviewed By: simpkins
Differential Revision: D7956732
fbshipit-source-id: 85d9f3fd725853a54da9e70fc659bd7eb9e0862c
Summary:
Add a function for diffing two source control commits without needing to
instantiate TreeInode objects.
Reviewed By: wez
Differential Revision: D7341604
fbshipit-source-id: 557eef87faa2785ab96d51b09569a46f892a71f6
Summary:
This removes the TARGETS files from the eden github repository. The
open source buck build has been failing for several months, since buck
removed support for the thrift_library() rule.
I will potentially take a stab at adding CMake build support for Eden
at some point in the future.
Reviewed By: chadaustin
Differential Revision: D6893233
fbshipit-source-id: e6023094a807cf481ac49998c6f21b213be6c288
Summary:
This is the type of a tree entry, which may be another tree, so
FileType is not an accurate name.
Reviewed By: simpkins
Differential Revision: D6981168
fbshipit-source-id: 997eb8a27f599310ed678ce221c8083722db8bff
Summary:
I want to rename FileType to TreeEntryType so I removed this one first
and replaced all of its uses with an isTree() method.
Reviewed By: simpkins
Differential Revision: D6980501
fbshipit-source-id: 105b8c599585e63efd44043e761db40e2824e77e
Summary:
Our Model TreeEntry code was a bit too general - in reality, both git
and hg only support a handful of specific tree entries: regular files,
executable files, symlinks, and trees. (git also supports
submodules.) This diff delays the expansion of a TreeEntry's type
into a full mode_t.
Reviewed By: simpkins
Differential Revision: D6980003
fbshipit-source-id: 73729208000668078a180b728d7e0bb9169c6f3c
Summary:
refactor this test so that we can apply it to any LocalStore
implementation, and have it run against the Memory, Rocks and SQLite
implementations.
Reviewed By: chadaustin, simpkins
Differential Revision: D6919455
fbshipit-source-id: cc93042b95833b175955e6395c84cf41238a90d2
Summary:
This enables dropping in alternative implementations
of LocalStore and adds a MemoryLocalStore implementation for
use in our tests.
This diff doesn't change the default storage option for the
eden server. I'll look at adding such an option in a follow up diff.
Reviewed By: chadaustin
Differential Revision: D6910413
fbshipit-source-id: 018bf04e0bff101e1f0ab35e8580ca2a2622e5ef
Summary:
This is a codemod to change from using @/ to // in basic cases.
- TARGETS files with lines starting with @/ (but excluding @/third-party:
- autodeps lines in source and TARGETS files ( (dep|manual)=@/ ), excluding @/third-party
- Targets in string macros
The only thing left of the old format should be @/third-party:foo:bar
drop-conflicts
Reviewed By: ttsugriy
Differential Revision: D6605465
fbshipit-source-id: ae50de2e1edb3f97c0b839d4021f38d77b7ab64c
Summary:
Per discussion with bolinfest, this brings Eden in line with clang-format.
This diff was generated with `find . \( -iname '*.cpp' -o -iname '*.h' \) -exec bash -c "yes | arc lint {}" \;`
Reviewed By: bolinfest
Differential Revision: D6232695
fbshipit-source-id: d54942bf1c69b5b0dcd4df629f1f2d5538c9e28c
Summary:
Add LocalStore::WriteBatch class so that LocalStore users
can manage own batches independently. This makes it safer to
perform concurrent batched updates as the current implementation
assumes (incorrectly) that it has exclusive ownership of the batch mode setting.
Reviewed By: simpkins
Differential Revision: D5911697
fbshipit-source-id: 62005245ce2542b87eac206a15d8657bcca81d83
Summary:
We're running down a performance problem with hg status that we believe
is something happening at a higher level in the code but noticed that there
were a lot of reads of the rocksdb SST files. In the strace output for those
we observed file content data being read. The status operation shouldn't
need file contents; what's happening is that we're over-fetching some metadata
but happen to be scooping up the file contents from the SST file because we
use the same key prefixes and differentiate the keyspace with key suffixes.
This diff implements the use of rocksdb column families to partition things
more effectively and results in a speed up of around 6x in this scenario.
Furthermore, applying point lookup optimization options yields an additional
2x performance improvement to our rocksdb performance.
As part of this diff, I've removed the hash set that we were using to allow
checking whether a key was present in the store; it wasn't very useful
and would have had to be split into one set per keyspace with this diff;
easier to just remove it.
Reviewed By: bolinfest
Differential Revision: D5781906
fbshipit-source-id: 97f068ade546fd09f391e60a7a57fec0e9081e67
Summary: It doesn't need to exist anymore
Reviewed By: yfeldblum
Differential Revision: D5318746
fbshipit-source-id: c70b184f4b3fc12ede4632d6b3d43de16ed758c7
Summary:
Format all of the TARGETS files under eden/fs with the autodeps tool.
A few rocksdb include statements require comments so that autodeps can
correctly tell which dependency this include comes from. The rocksdb library's
source file structure unfortunately does not match the layout of how its header
files get installed, so autodeps cannot figure this out automatically.
Reviewed By: wez
Differential Revision: D5316000
fbshipit-source-id: f8163adca79ee4a673440232d6467fb83e56aa10
Summary:
Remove the ObjectStores.h and .cpp files. These files contained helper
functions for looking up Tree and TreeEntry objects based on a deep relative
path.
These functions are no longer used anywhere anymore (the code that performs
deep lookups always does inode lookups instead). Additionally, this file is
the only place still using some of the blocking, non-Future-based APIs for
object lookups. Deleting these files will let us remove these deprecated
blocking APIs.
Reviewed By: wez
Differential Revision: D5295691
fbshipit-source-id: 89229827305490eba4d3a581954a90c5cf63238d
Summary:
This updates all of the references to gtest and gmock with googletest.
The change is mechanilcal, generated with the following one-liner:
```lang=bash
hg grep -lwE '(gtest|gmock)' 'glob:**/TARGETS' | grep -v '^third-party-buck' | xargs perl -pi -e '
$gt=qr!(["'"'"'])gtest\g1!;
(
s!$gt(\s*,\s*(.any.|None))(\s*,\s*)?\),?!\1googletest\1\2, \1gtest\1\),!g or
s!$gt((\s*,\s*(.any.|None)[^\)]+))\),?!\1googletest\1\2\),!g or
s!\(\s*$gt,?\s*\),?!\(\1googletest\1, None, \1gtest\1\),!g or
s!$gt,?!\(\1googletest\1, None, \1gtest\1\),!g
) unless /(name|type) *=/;
$gm=qr!(["'"'"'])gmock\g1!;
(
s!$gm(\s*,\s*(.any.|None))(\s*,\s*)?\),?!\1googletest\1\2, \1gmock\1\),!g or
s!$gm((\s*,\s*(.any.|None)[^\)]+))\),?!\1googletest\1\2\),!g or
s!\(\s*$gm,?\s*\),?!\(\1googletest\1, None, \1gmock\1\),!g or
s!$gm,?!\(\1googletest\1, None, \1gmock\1\),!g
) unless /(name|type) *=/;
'
```
Reviewed By: meyering
Differential Revision: D4643237
fbshipit-source-id: fda7f41760c7e44254231df87634631c343e6355
Summary:
Remove AutoHeaders.RECURSIVE_GLOB from all of our TARGETS files. This is a
Facebook-internal directive that was added as part of a codemod. The vanilla
open source buck does not know about this directive, and can't parse TARGETS
files that contain it.
All of the affected rules are cpp_unittest() and cpp_binary() rules that don't
actually have any header files, so these parameters didn't actually have any
effect.
Reviewed By: andrewjcg
Differential Revision: D4633748
fbshipit-source-id: 6227fe9b7f2ea838a0c72408ca195685bcae9f5a
Summary:
Update copyright statements to "2016-present". This makes our updated lint
rules happy and complies with the recommended license header statement.
Reviewed By: wez, bolinfest
Differential Revision: D4433594
fbshipit-source-id: e9ecb1c1fc66e4ec49c1f046c6a98d425b13bc27
Summary:
Rename the existing TestBackingStore class to FakeBackingStore, and fill it out
with an implementation that allows test code to control the store.
The test code can populate the store with Trees and Blobs to return, and can
control when the Futures returned by the store are fulfilled.
Reviewed By: bolinfest
Differential Revision: D4338577
fbshipit-source-id: 79221b04d844bd6011078b799e55182de4ccdfdc
Summary:
This updates the EdenServer and LocalStore classes to require more arguments be
passed in as AbsolutePath arguments rather than just plain strings.
This updates the main program to process path arguments using canonicalPath().
Reviewed By: bolinfest
Differential Revision: D4332273
fbshipit-source-id: 3d235a767963b11129c3897ad027ad761b6dae50
Summary: These should have been added as part of D4270526.
Reviewed By: simpkins
Differential Revision: D4315382
fbshipit-source-id: 5920ff38f9cc63540e4813e8ab40f79ad46f9ec1
Summary:
Move the FakeObjectStoreTest class into fs/testharness, along with the
TestMount and TestBackingStore classes. This simply consolidates the test
utility code into a single location.
Reviewed By: bolinfest
Differential Revision: D4317517
fbshipit-source-id: 4e19590c5ffde88b66f2c8d4a964352ec349031c
Summary:
Hash objects are small enough (20 bytes) that it isn't worth allocating them on
the heap. This updates LocalStore::getSha1ForBlob() to return a
folly::Optional<Hash>, and ObjectStore::getSha1ForBlob() to return a plain
Hash.
Reviewed By: bolinfest
Differential Revision: D4298162
fbshipit-source-id: 9cf54f2997ba8c3b2346db315a2aca41e580b078
Summary:
Add comments in ObjectStore.h documenting the fact that the get* APIs all throw
std::domain_error when the specified ID does not exist, and never return
nullptr.
Also update the FakeObjectStore class used for testing to follow this behavior.
Reviewed By: bolinfest
Differential Revision: D4298160
fbshipit-source-id: c5509bb3aa2ed76619b06b733ad240aaa5f00862
Summary:
In addition to storing the SHA-1 of each file's contents, also store the size.
This will allow us to more quickly look up the file size, without having to
retreive the file size.
I haven't yet added an API to ObjectStore to retreive the full BlobMetadata
object; I will do that in a subsequent diff. One benefit for now is that this
does avoid double-computing the SHA-1 in ObjectStore::getSha1ForBlob() if we
had to load the blob.
Reviewed By: bolinfest
Differential Revision: D4298157
fbshipit-source-id: 4d83ebfa631c93fcef06ca1cd0ba0e1a70a2476d
Summary:
This is analogous to the existing `getEntryForFile()` helper function that we
have, and I was able to rewrite `getEntryForFile()` in terms of
`getTreeForDirectory()`, which simplifies the code considerably.
Also moved things from `eden/fs/model/hg/misc.h` to
`eden/fs/store/ObjectStores.h`, which is much more appropriate.
Reviewed By: wez
Differential Revision: D4032817
fbshipit-source-id: ff4d32120fb050f8b5c5c53b7f2e94b524781648
Summary:
In an upcoming revision, I am going to introduce a utility function that takes
an `ObjectStore` (well, now an `IObjectStore`) as a parameter and I want to be
able to test it. Having a `FakeObjectStore` should make this considerably easier
without having to resort to mocks.
Reviewed By: simpkins
Differential Revision: D3980580
fbshipit-source-id: 5886e2055c893e749cc898226e1baade776c3ea7
Summary:
This codemods `TARGETS` under `[a-d]*` directories in fbcode to make
the `headers` parameter explicitly refer to `AutoHeaders.RECURSIVE_GLOB`.
Reviewed By: yfeldblum
Differential Revision: D3801845
fbshipit-source-id: 715c753b6d4ca3a9779db1ff0a0e6632c56c0655
Summary: This change updates LocalStore to perform serialization of trees and blobs internally so that its users don't need to be aware of the internal serialization format. Previously, the get and put APIs were asymmetric such that the get APIs returned deserialized Tree and Blob objects, while put required raw serialized bytes. After this change, put will also use deserialized Tree and Blob objects.
Reviewed By: simpkins
Differential Revision: D3589899
fbshipit-source-id: 2e572e6ec5af44d66206b178a03f7a9d619b2290
