Summary:
Similarly to the previous diff, reducing the lock scope will improve
concurrency leading to higher performance in EdenFS.
Reviewed By: andll
Differential Revision: D30595787
fbshipit-source-id: 1d52e4a8d362f7e2e3e18c2a57a3ebb7628f549e
Summary:
Similarly to the previous diff, let's not hold any read/write locks when not
needed. This will improve concurrency of the code.
Reviewed By: andll
Differential Revision: D30595786
fbshipit-source-id: 6ea6c689e4deca713051a9f3611647334c528bc7
Summary:
In the context of EdenFS, during a prefetch operation, EdenFS will call into
the revisionstore (via the backingstore) from multiple threads and use the
ContentStore::prefetch method in each of them with batches of keys.
Unfortunately, the IndexedLogDatastore lock is write held both when serializing
fetched data and when writing to the underlying IndexedLog. The serialization
part can be fairly CPU intensive as it will compress data with LZ4, but doesn't
require the lock to be held. If a concurrent get_missing call to the
IndexedLogDatastore is made, that will thus wait for the compression to
complete, reducing overall concurrency and fetching speed.
Reviewed By: andll
Differential Revision: D30583566
fbshipit-source-id: 06f5f4988c1bc911ae155189317232b54915a5cf
Summary: Add a new `BackingStore` implementation based on `scmstore` instead of `ContentStore` and allow the backend to be selected with the `scmstore.backingstore` config.
Reviewed By: andll
Differential Revision: D29672974
fbshipit-source-id: dc6b8662903bcfc941b586544aad487de6b8c956
Summary: Introduce a new `StoreValue` trait, implemented for `StoreFile`, for use in `CommonFetchState`, which will be shared between `TreeStore` and `FileStore`.
Reviewed By: kulshrax
Differential Revision: D30295987
fbshipit-source-id: 862349980befa13cb3d420ee617e4fe5464d890a
Summary: Introduce a new `StoreAttrs` trait, implemented for `FileAttributes`, for use in `CommonFetchState`, which will be used by both `TreeStore` and `FileStore`.
Reviewed By: kulshrax
Differential Revision: D30295969
fbshipit-source-id: 603242fe08bea72035b1b1d0f37c46514e1fb92b
Summary: Refactoring in preparation for updating `TreeStore` to match `FileStore`
Reviewed By: andll
Differential Revision: D30295957
fbshipit-source-id: 0f1677311eb2578d8e0dae12c07a1599edc3b500
Summary: Making `FetchResults` generic in preparation for using it with trees too.
Reviewed By: andll
Differential Revision: D30286592
fbshipit-source-id: a48cf2dbcdcc2c4b8a102eaa02ac465c367c6793
Summary: I'll be adding metrics to the `TreeStore` soon, so I'm refactoring code that will be shared between files and trees. This change moves the non-file-specific metrics types up to the top-level `scmstore` module, rather than inside `scmstore::file`.
Reviewed By: andll
Differential Revision: D30284933
fbshipit-source-id: 31f48312bca8d75d4893220cd189b9735a37a5a0
Summary: Helps figure out what happens to metalog internally.
Differential Revision: D30563249
fbshipit-source-id: 10323d36d762edda93206dd01c88d1f0d8abdf8d
Summary:
The failpoint feature supports more complex injections, such as sleep, or fail
after a few times. There is no need to keep the adhoc faultinjection feature.
Differential Revision: D30495223
fbshipit-source-id: b5613811e489a5a52e9c0dd1ebf1096c848a402b
Summary: Currently, tree imports are queued regardless of whether they are in the `hgcache`. This adds unnecessary delay, especially if the queue is busy (importer takes a long time and causes queue to backlog). This diff adds the logic to check if the tree is in `hgcache` before enqueuing a tree import request.
Reviewed By: xavierd
Differential Revision: D30514871
fbshipit-source-id: eb23f64b7f059832571f957fb67d18c3821d2844
Summary:
this library is a more general version of the `panic_unpack` lib I
made in fbcode. I made this library, its mit-apache licensed
Reviewed By: dtolnay
Differential Revision: D30607308
fbshipit-source-id: ee4fad3924fdae021753055cd3fd88c99cb99512
Summary:
This allows us to insert FAILPOINTS in Python so we can use sleep, return error
etc.
Differential Revision: D30495224
fbshipit-source-id: aef56d03bc32eefb69573cfa586aa63a301edffc
Summary:
I abandoned D30603353 in favor of this one because
its_cleaner
We don't need repo name in every hook, just 2 of them.
This will allow us to have predefined, named hook configs that are repo-agnostic. E.g. when repositories are similar, they could share one set of hook configs in configerator.
There were two hooks that had repo_name in configerator hook config: verify_integrity and verify_reviewed_by
Reviewed By: StanislavGlebik
Differential Revision: D30605229
fbshipit-source-id: c310b16b564808d0dc0909d21cc3521a57e06fad
Summary: The result is too flakey now that we've introduced threading.
Reviewed By: andll
Differential Revision: D30607733
fbshipit-source-id: f8bfa2a57d427731fb4ac3011f4364190a83b771
Summary:
Some code in the HgDatapackStore is overly complicated due to the fact that
revHash returns a owned Hash and this forces the code to thus copy it onto a
temporary vector. By having a method that can directly return a slice to the
hash, this issue disappears, thus let's add it.
Reviewed By: chadaustin
Differential Revision: D30582458
fbshipit-source-id: dc102117bc82ab72378293c0abfe9acfd862e9e6
Summary: Cleaned up all remaining usages of this deprecated API in CTP codebase
Differential Revision: D30517771
fbshipit-source-id: 6b2c7fb6c569bf5a928a7eec60fdd890baad312f
Summary:
Handling of mutable renames was incorrect for two reasons:
1) We didn't add an entry to history graph, so only a single changeset before
rename was returned. That was easy to "fix" (just add a new entry to history
graph), but...
2) ...all history operations now have to use a different path (the source of
the rename path).
To fix it let's track not just the changeset id, but also the path for the
given changeset id. Since the path can potentially be large I wrapped it into
Arc to avoid expensive clones.
Differential Revision: D30576342
fbshipit-source-id: a99f6269c34b0a0c626104ec47c9392f984328fb
Summary:
This diff calls the `/:repo/snapshot` EdenApi endpoint added on D30514854 (ab17c4d181) from the `hg snapshot restore` command.
For now, it just prints the parent of the snapshot, but in next diffs it will update to it and restore the dirty changes.
Reviewed By: StanislavGlebik
Differential Revision: D30517984
fbshipit-source-id: e1381eaed561a7184ee02ab99d0282f11a1d944f
Summary: This diff adds the `fetch_snapshot` method to the EdenApi trait, implements it for talking with the EdenApi service, and also adds python bindings for ease of use from python.
Reviewed By: StanislavGlebik
Differential Revision: D30517973
fbshipit-source-id: 41c24ba25040b397b7d739c2885a47acfb9100d2
Summary:
I noticed that every time we fetch blob from hg, we calculate sha hash and put it into metadata table.
Both calculating sha1 of content and writing it to rocks is fairly expensive, and it would be nice if we can skip doing so in some cases.
In this diff I use inexpensive cache check to see if we already calculated metadata for given blob and skip recalculation
In terms of performance, it reduces blob access time in hot case from **0.62 ms to 0.22 ms**.
[still need to do some testing with buck, but I think this should not block the diff since it seem farily trivial]
This is short-medium term fix, the longer term solution will be keeping hashes in mercurial and fetching them via eden api, but this will take some time to implement
Reviewed By: chadaustin, xavierd
Differential Revision: D30587132
fbshipit-source-id: 3b24ec88fb02e1ea514568b4e2c8f9fd784a0f10
Summary: Similarly to the enqueue benchmark, let's have a dequeue benchmark.
Differential Revision: D30560489
fbshipit-source-id: ae18f7e283e4bab228aaa0f58bff2e6f2cfa3021
Summary:
In order to enqueue and find an element in a hash table, the key needs to be
hashed. Hashing a HgProxyHash relies on hashing a string which is significantly
more expensive than hashing a Hash directly. Note that they both represent the
same data and thus there shouldn't be more collisions.
Reviewed By: chadaustin
Differential Revision: D30520223
fbshipit-source-id: 036007c445c28686f777aa170d0344346e7348b0
Summary:
Allocations are expensive, especially when done under a lock as this increase
the critical section, reducing the potential concurrency. While this yields to
a 1.25x speedup, this is more of a sideway improvement as the allocation is now
done prior to enqueuing. This also means that de-duplicating requests is now
more expensive, as no allocation would be done before, but at the same time,
de-duplication is the non-common code path, so the tradeoff is worthwhile.
Reviewed By: chadaustin
Differential Revision: D30520228
fbshipit-source-id: 99dea65e828f9c896fdfca6b308106554c989282
Summary: The F14 hashmap are significantly faster than the std::unordered_map.
Reviewed By: chadaustin
Differential Revision: D30520225
fbshipit-source-id: d986908c5eac17f66ae2c7589f134c430a3c656e
Summary:
When turning on the native prefetch, EdenFS will enqueue tons of blob requests
to the import request queue. The expectation is then that the threads will
dequeue batch of requests and run them. What is being observed is however
vastly different: the dequeued batches are barely bigger than 10, far lower
than the batch capacity, leading to fetching inefficiencies. The reason for
this is that enqueuing is too costly.
The first step in making enqueuing less costly is to reduce the number of times
the lock needs to be acquired by moving the de-duplication inside the enqueue
function itself. On top of reducing the number of times the lock is held, it
also halves the number of allocation done under the lock.
Reviewed By: chadaustin
Differential Revision: D30520226
fbshipit-source-id: 52f6e3c1ec45caa5c47e3fd122b3a933b0448e7c
Summary:
It turns out that we do want to use a Future to make sure that the tracebus and
watches are completed on the producer and not on the consumer of the future. We
could use a `.via(inline executor)` but the code becomes less readable, so
let's just revert the diff.
Reviewed By: chadaustin
Differential Revision: D30545721
fbshipit-source-id: 524033ab4dbd16be0c377647f7f81f7cd57c206d
