Summary:
There are a lot of hash types: HgId, Sha256, and edenapi-types has some.
The HgId seems to have most features but it is hard to reuse it for similar
but different types. This is an attempt to do so and unify the main implementation
of those types.
Most methods are copied from HgId. `from_byte_array` is made const fn so it
can be used to construct constants.
The error type is intentionally chosen to be not `anyhow::Error`.
Using static typed errors is considered good practice for low-level crates.
The benefit is that higher level users get their choice - precise static error
type with compile-time checks, or convenient, dynamic error types by anyhow
with runtime downcasts.
Reviewed By: yancouto
Differential Revision: D31615719
fbshipit-source-id: 337356721354c43fe23b9f2d0e90d104c8864c44
Summary:
It's not actually used and is hard to integrate with hg now.
`bookmarkstore` was initially designed before metalog (provides atomicity for
metadata changes) being a thing.
With metalog we track bookmarks using plain text strings, complex structures
like indexedlog cannot be used within metalog.
Reviewed By: yancouto
Differential Revision: D31615721
fbshipit-source-id: 75d6b9c9ba4475e86530e2368d30f53e58ac37d8
Summary:
See previous diffs for context. This makes the test shorter and will eventually
allow us to remove some bloated code in edenapi.
Reviewed By: yancouto
Differential Revision: D31465811
fbshipit-source-id: 2c0c0bab4adc4fb89fe9b46bc3f204c79efe17ee
Summary:
See previous diffs for context. This makes the test shorter and will eventually
allow us to remove some bloated code in edenapi.
Note some hashes are serialized (suboptimally) as an array of bytes, not as a
byte slice. Ideally we need `serde_bytes` annotation somewhere but I haven't
checked if that breaks compatibility.
Reviewed By: yancouto
Differential Revision: D31465828
fbshipit-source-id: 7892ececc475bad530708499cf2255852d610bc2
Summary: They will be used by `debugapi`.
Reviewed By: yancouto
Differential Revision: D31615722
fbshipit-source-id: b3e53a8ac0ab8202905602b478c1ca9e4d712f64
Summary:
These are used by Mononoke tests. Expose them so `debugapi` can be used in
tests.
Reviewed By: yancouto
Differential Revision: D31465832
fbshipit-source-id: 4291023d5d4623c43065e1b1c90848f3bc15047f
Summary:
See previous diffs for context. This makes the test shorter and will eventually
allow us to remove some bloated code in edenapi.
Note the client-side edenapi only provides prefix lookup. So it does not work
for an arbitrary range.
Reviewed By: yancouto
Differential Revision: D31465812
fbshipit-source-id: 9b81148258c45e7e534faee97e1b51c4dd75102d
Summary:
See previous diffs for context. This makes the test shorter and will eventually
allow us to remove some bloated code in edenapi.
Reviewed By: yancouto
Differential Revision: D31465827
fbshipit-source-id: 3e7a943efce6c670d817f57e10a1e1e191fe633f
Summary:
See previous diff for context. This provides pure Rust HgId more easily (without
needing `py`), and makes the acceptable format a bit more flexible, useful for
`debuapi`.
Unfortunately `Serde<Key>` or `Serde<Vec<(RepoPathBuf, HgId)>>` is not really
a choice because `RepoPathBuf` did not declare as `#[serde(transparent)]`,
which means paths have to be written as `(path,)`. Adding `#[serde(transparent)]`
to `RepoPathBuf` might be a breaking change that I didn't try.
Reviewed By: yancouto
Differential Revision: D31465819
fbshipit-source-id: cda210c2f5f6532256204abd428e1ad2b1de9fd9
Summary:
These APIs wrote directly to stores. Change them to return content like other
APIs so `debugapi` can print the result.
As we're here, drop legacy progress support.
Reviewed By: yancouto
Differential Revision: D31465820
fbshipit-source-id: 0c7a2e07f8fe56a89cc82a51ca6566d3ab6cc754
Summary:
D31057140 (0cd8e64c51) removed some serde derives on structs, but we need to them for easier
Python bindings. Let's add them back.
Reviewed By: yancouto
Differential Revision: D31465804
fbshipit-source-id: 94e8f0dbbbde4e52cb8b3153ea219fa70c91fca4
Summary:
See previous diffs for context. This makes the test shorter and will eventually
allow us to remove some bloated code in edenapi.
Note the client-side edenapi only provides prefix lookup. So it does not work
for an arbitrary range.
Reviewed By: yancouto
Differential Revision: D31465834
fbshipit-source-id: eb45922e109b7301beb9799e3ccb7905541de605
Summary:
See previous diffs for context. This makes the test shorter and will eventually
allow us to remove some bloated code in edenapi.
Reviewed By: yancouto
Differential Revision: D31465823
fbshipit-source-id: c2a89cbac62ae6d4aa80bde41c177bc4acd986fa
Summary:
See previous diffs for context. This makes the test shorter and will eventually
allow us to remove some bloated code in edenapi.
Reviewed By: yancouto
Differential Revision: D31465813
fbshipit-source-id: b921476f2d398271688ee96cc994474b616358f0
Summary:
See previous diffs for context. This makes the test shorter and will eventually
allow us to remove some bloated code in edenapi.
Reviewed By: yancouto
Differential Revision: D31465818
fbshipit-source-id: 7885d7204d01a5ae7a0e835eeb9e51cedcc6281d
Summary:
See previous diffs for context. This makes the test shorter and will eventually
allow us to remove some bloated code in edenapi.
Reviewed By: yancouto
Differential Revision: D31465815
fbshipit-source-id: 052a8a4487793a6a977e8ff27aa2bc5443175061
Summary:
This is not used in production and the removal unblocks the removal
of the `complete_trees` endpoint.
Reviewed By: yancouto
Differential Revision: D31465807
fbshipit-source-id: d2a8ff79fe4e6181adefa499bbd7125fa0c5c26b
Summary:
See previous diffs for context. This makes the test shorter and will eventually
allow us to remove some bloated code in edenapi.
Reviewed By: yancouto
Differential Revision: D31465798
fbshipit-source-id: 8e184a881479e42fcda06d7258af55fab99ab1da
Summary:
`debugapi` was introduced to test edenapi endpoints as seen by `hg`. Use it to
test the real client-side edenapi logic. This will eventually allow us to drop
`read_res`, `make_req`, `FromJson`, and `ToJson` to reduce bloated code in
edenapi.
Reviewed By: yancouto
Differential Revision: D31465799
fbshipit-source-id: 5a1ae0565b9c7a23773644e775c5f1930ebc12ad
Summary: It was a temporary step that is never used in production. Delete the endpoint.
Reviewed By: yancouto
Differential Revision: D31465808
fbshipit-source-id: 90e77eab96bb75796a3b31a5907f137d743a7dfa
Summary: It was a temporary step that is never used in production. Delete the endpoint.
Reviewed By: yancouto
Differential Revision: D31465829
fbshipit-source-id: 635ee205589b0f4d15388ae2a2bb9ada51d77edd
Summary: This makes it a bit more efficient and more friendly in debugapi output.
Reviewed By: DurhamG
Differential Revision: D31465800
fbshipit-source-id: b741792eb43a7a57c90f75362dbf189739dbb844
Summary:
`debugapi` was introduced to test edenapi endpoints as seen by `hg`. Use it to
test the real client-side edenapi logic. This will eventually allow us to drop
`read_res`, `make_req`, `FromJson`, and `ToJson` to reduce bloated code in
edenapi.
Reviewed By: yancouto
Differential Revision: D31465830
fbshipit-source-id: 9da739a76ef6e5d49804b0cea2089fc1741d0b7c
Summary:
`debugapi` was introduced to test edenapi endpoints as seen by `hg`. Use it to
test the real client-side edenapi logic. This will eventually allow us to drop
`read_res`, `make_req`, `FromJson`, and `ToJson` to reduce bloated code in
edenapi.
Reviewed By: yancouto
Differential Revision: D31465814
fbshipit-source-id: 5d090e0a92c6374b15a36f80dcc578ad280e50e2
Summary: To ease review, just doing the simple / hacky version of the TreeStore diff, like the FileStore one below this. This doesn't address the tree batch failure issue, but that's not blocking release so it's fine for now.
Reviewed By: andll
Differential Revision: D31593076
fbshipit-source-id: 0d3c420e50af0d8882ba171590597aac1b6c4c77
Summary:
Currently, the scmstore backingstore backend for EdenFs performs very poorly compared to ContentStore. This is mainly because a local-only FileStore is created on every fetch, and always flushes when dropped (even when not used).
With this change, we'll only flush when the parent FileStore is dropped.
This is a slightly hacky fix, vs. creating a new non-flush-on-drop type like `TreeStoreFetch` as we did for trees in the previous diff, but this should work fine for now and is radically smaller. Later I'd like to clean up both to eliminate the verbosity in the trees approach while still using separate types rather than a bool.
Reviewed By: andll
Differential Revision: D31591276
fbshipit-source-id: 11266c19ac68d87015719f5bcbd8f857e596bfdb
Summary: With edenapi and memcache reads both being written to disk as they're fetched, we should no longer need to break large prefetch batches into chunks.
Reviewed By: DurhamG
Differential Revision: D31552341
fbshipit-source-id: 2d9e10db669754cc8124228252cf832c8102f220
Summary: Like with the previous EdenApi change, writing files fetched from memcache to disk as we read them will prevent us from accumulating large amounts of file content in memory and potentially causing an OOM.
Reviewed By: DurhamG
Differential Revision: D31552190
fbshipit-source-id: 1eceb7570918575382f067ff9ad1e08e0623f335
Summary: Introduce a new method, `evict_to_cache`, which writes a memory-backed LazyFile to disk and returns a mmap-backed LazyFile instead.
Reviewed By: DurhamG
Differential Revision: D31551177
fbshipit-source-id: 86901628c56151ac805a57885379241c42f794c0
Summary:
Use the new async `files_attrs` method to concurrently fetch files from EdenApi, write them to disk, and read them back as a mmap-ed `LazyFile`.
Also remove scmstore prefetch chunking with the previous approach required.
Previously, we fetched all the requested files from EdenApi into memory as a batch, then wrote them to disk afterward. This caused OOMs for extremely large fetches, and required chunking scmstore prefetches, which had a negative performance impact.
Reviewed By: DurhamG
Differential Revision: D31445678
fbshipit-source-id: 9a2e1476fb8ddfcd546a5e0b501cc91cc2a97303
Summary: Add a basic async `files_attrs` method to `EdenApiFileStore` for use by scmstore for concurrent fetching / writing to disk.
Reviewed By: DurhamG
Differential Revision: D31483490
fbshipit-source-id: 20233767541d60ccdfcd97bdc98b9bbb98d8700e
Summary:
Currently, scmstore does not write LFS pointers discovered outside the LfsStore before fetching blobs from the LFS remote and attempting to read them back from disk. This causes scmstore to rely on the contentstore fallback to write LFS pointers, after which future reads will succeed in scmstore.
With this change, we track which LFS pointers were discovered from a source other than an LfsStore, and write those to the LfsStore before fetching from the LFS remote. This eliminates the debugimporterhelper fallback in the backingstore implementation, and should improve the performance of the shim in general for LFS files where the LFS pointer is not available locally.
In the future, I'll probably perform these writes inline with the fetches that discovered the pointers for added concurrency, and reorganize the LFS fetching code in general to avoid some of the weird bookkeeping we currently do around local vs. shared for LFS.
Reviewed By: andll
Differential Revision: D31280512
fbshipit-source-id: 2e2a32db3fc9fab8ac179b7852da36f77c66ef31
Summary: we run the eden endpoints from mononoke server, not via this binary any more. remove it
Reviewed By: StanislavGlebik
Differential Revision: D31501592
fbshipit-source-id: 0626fe43f0f1ce4a6c7165a734eb487225fa65b6
Summary: Derived data manager doesn't need full BlobRepo struct. So lets create facet container which includes only needed parts for derivation using manager. I'll use it later with 2DS service.
Reviewed By: StanislavGlebik
Differential Revision: D31540177
fbshipit-source-id: 42a3e96352689f39730b8f62142cccd7a1bb7256
Summary: I'm going to use that API from 2DS service. so the discovery of underived will not be triggered.
Reviewed By: yancouto
Differential Revision: D31540178
fbshipit-source-id: cdf91974b90bc07e09cd3f465f9a1f8b8271a8e4
Summary:
Background: I've been looking into derived data performance and found that
while overall performance is good, it depends quite a lot on the blobstore
latency i.e. the higher the latency the slower the derivation. What's worse is
that increasing blobstore latency even by 100ms might increase time of
derivation of 100 commits from 12 to 65 secs! [1]
However we have ways to mitigate it:
* **Option 1** If we use "backfill" mode then it makes derived data derivation less
sensitive to the put() latency
* **Option 2** If we use "parallel" mode then it makes derived data derivation less
sensitive to the get() latency.
We can use "backfill" mode for almost all derived data types (only exception is
filenodes), however "parallel" only enabled for a few derived data types (e.g.
fsnodes, skeleton manifests, filenodes).
In particular, we didn't have a way to do batch derived data derivation for
unodes, and so unodes derivation might get quite sensitive to the blobstore
get() latency. So this diff tries to address that.
I considered three options:
* **Option 1** The simplest option of implementing "parallel" mode for unodes is to just
do a unode warmup before we start a sequential derivation for a stack of commits. After the
warmup all necessary entries should be in cache, so derivation should be less latency sensitive.
This could work, but it has a few disadvantages, namely:
* We do additional traversal - not the end of the world, but it might get
expensive for large commits
* We might fetch large directories that don't fit in cache more often than we
need to.
That said, because of it's simplicity it might be a reasonable option to keep
in mind, and I might get back to it later.
* **Option 2** Do a derivation for a stack of commits. We have a function to derive a
manifest for a single commit, but we could write a similar function to derive the whole stack at once.
That means for each changed file or directory we generate not a single change
but a stack of changes.
I was able to implement it, but the code was too complicated. There were quite
a few corner cases (particularly when a file was replaced with a directory, or
when deriving a merge commit), and dealing with all of them was a pain.
Moreover, we need to make sure it works correctly in all scenarios, and that
wouldn't be an easy thing to do.
* **Option 3** Do a derivation for a "simple" stack of commits. That's basically the
simplified version of option #2. Let's allow doing batch derivation only for
stacks that have no
a) merges
b) path changes that are ancestors of each other (which cause file/dir
conflicts).
This implementation is significantly simpler than option #2, and it should
cover most of the cases and hopefully bring perf benefits (though this is
something I'm yet about to measure). So this is what this diff implements
Reviewed By: yancouto
Differential Revision: D30989888
fbshipit-source-id: 2c50dfa98300a94a566deac35de477f18706aca7
Summary: I'd like to reuse it in the next diff, so let's move it to a separate function
Reviewed By: yancouto
Differential Revision: D31603911
fbshipit-source-id: 69ec0553022aabe662f75a50321423c54aafd196
Summary:
We don't really need it, and InnerRepo is more expensive to create, so let's
not do that.
Reviewed By: yancouto
Differential Revision: D31602266
fbshipit-source-id: 269bf50f8b4ee99e22888d91af2ac392078d32fa
Summary:
We were `hg adding` all files that were modified, however, we only really need to add files that didn't exist before the snapshot changes.
This gets rid of the annoying "already tracked!" messages.
Reviewed By: markbt
Differential Revision: D31438120
fbshipit-source-id: ca3545bc5881dcf01283abfa5ec9eca6309ff607
Summary: updated library.sh timeout handling to be clearer by checking the time change directly, and to use the health_check endpoint
Reviewed By: yancouto
Differential Revision: D31536407
fbshipit-source-id: 43cdf4260c10dfd4d3097dcd92d071c3d18b18f8
Summary:
We don't actually use bundle2 to deliver manifests anymore. Everything
is done via ondemand fetching, which won't have the problem covered in this
test. So let's delete it.
Reviewed By: quark-zju
Differential Revision: D31309313
fbshipit-source-id: 312508fa1b5e903314b92c048d23525c2194ab91
Summary:
This is part of removing filepeer. I also enabled treemanifest and
modernclient (i.e. lazy changelog) on a few tests.
Reviewed By: quark-zju
Differential Revision: D31032055
fbshipit-source-id: 6822274ad07303ed86b2ee5dd4e09979f1e215d5
Summary:
This test covers 'hg clone -r REV' which isn't really a supported clone
case since we now depend on cloning particular remote bookmarks. The test is
also heavily dependent on revision numbers. So let's just delete it.
This is needed as part of removing the tests dependency on hg server logic like
filepeer.
Reviewed By: quark-zju
Differential Revision: D31309312
fbshipit-source-id: e4620186e3eda3114686de36d06710747439ae18
Summary:
Ports test-bundle.t to use modernclient configs, meaning it no longer
uses the filepeer and it uses treemanifest and lazychangelog. I deleted the
parts of the test that depend on mounting a bundle file on top of a repo, since
that logic is unused in production.
Reviewed By: quark-zju
Differential Revision: D31309310
fbshipit-source-id: a535ed9a21253fd258f70088e7436480957afb2a