Summary:
For norepo use-cases the config remains unset because dynamicconfig (still)
requires a repo. This means lazy changelog clone won't work. To workaround
it, we set `http.convert-cert` to `true` directly on Windows.
Reviewed By: andll
Differential Revision: D29804599
fbshipit-source-id: c0fc819711ec8d4f9f77cbce7b9d08b193ee9e6d
Summary:
Using all the preparations added in the stack, this diff adds the `/:repo/ephemeral/prepare` endpoint to eden api.
It simply creates an ephemeral bubble and returns its id via the call.
Reviewed By: markbt
Differential Revision: D29698714
fbshipit-source-id: 5bc289cad97657db850b151849784e50a17a9da6
Summary:
The memcache client has a 2+s setup time, so let's avoid it for short
commands.
Reviewed By: quark-zju
Differential Revision: D29627225
fbshipit-source-id: c755fbaadd35e423b6dafe772ffbed82fe41abce
Summary:
Previously we could add a MemcacheStore to the store hierarchy in two
ways: directly as MemcacheStore and through the MemcacheHgIdData/HistoryStore
wrappers.
In a future diff we'll want to track how long since the store was created and
not call the inner MemcacheStore in certain situations. To do this, we want all
accesses to go through the MemcacheHgId*Store abstraction.
Reviewed By: quark-zju
Differential Revision: D29627226
fbshipit-source-id: 9e979281fbb2eec123577d99a8879bcf80578136
Summary:
The memcache client takes 2 seconds to set up, so let's not use it for
short commands.
Reviewed By: quark-zju
Differential Revision: D29627229
fbshipit-source-id: bbefcd362e215a3b0f8a0f07c39b7ef00c71379e
Summary:
The previous diff only added these metrics to the scmstore path. Let's
add it to the legacy path as well, so we can start getting some insights while
scmstore rolls out.
Reviewed By: quark-zju
Differential Revision: D29601063
fbshipit-source-id: cb1935d02bc0b758c63abdf5d59f7a00d05ff4eb
Summary:
EspanId and tracing::span::Id had From/Into implementations that allowed them to be treated as
equivalent when they shouldn't be. This allowed for a subtle but where TracingData::record was
receiving an EspanId where the id was some huge u64 from a tracing::span::Id, when instead that
huge u64 should've been mapped to the real EspanId via the id_map.
Ideally we'd treat EspanId and the tracing Id separately, but the TracingCollector Subscriber
implementation relies on this. Since it controls span creation it can guarantee that all Id's are safely
convertible to EspanId.
Reviewed By: quark-zju
Differential Revision: D29599901
fbshipit-source-id: 79d24d41d86c6098888b1747cc0b9bc2838493fa
Summary:
This currently works well with ContentStore with EdenApi backend, did not test in other combinations, probably some other tweaks will be needed
Injection is controlled by MISSING_FILES env variable, it is interpreted as list of comma separated repo path prefixes
Reviewed By: DurhamG
Differential Revision: D29640090
fbshipit-source-id: 4925eabd63dc3a28a1133d332a072c0e224ea74b
Summary: This adds some types that will be used by the `/ephemeral/prepare` call in eden api. The call will actually be added on D29698714.
Reviewed By: liubov-dmitrieva
Differential Revision: D29696921
fbshipit-source-id: 9516661c1f41fcf87d37181fbab3eda5b6131179
Summary:
Clean up usage of callback or progress in the new code.
There is no need to use a callback or progress. They were used for legacy progress support. The new progress bar automatically applies to every HTTP request.
Reviewed By: yancouto
Differential Revision: D29710912
fbshipit-source-id: 33889d89680c90e63f4520626a166d0b39b67afb
Summary:
Remove the panicking APIs in D29647203 and instead propagate "missing content attribute" errors.
It was difficult to make these changes at the bottom of my diff stack, so I've added them here instead.
Reviewed By: DurhamG
Differential Revision: D29670495
fbshipit-source-id: 952ed4913a413c39ac3dff14a22f56e4766512ff
Summary: Add an option, `scmstore.prefercomputingauxdata`, which enables computing aux data from locally available content instead of requesting aux data from EdenApi when possible.
Reviewed By: DurhamG
Differential Revision: D29659777
fbshipit-source-id: e381c8beac359dc1735d76378c602fbf2bb0b668
Summary: Update `FileStore` to fetch aux data from EdenApi. As written, `FileStore` will prefer remotely fetching aux data to computing it from locally available content.
Reviewed By: DurhamG
Differential Revision: D29659721
fbshipit-source-id: 13e33830ed84fbba31e19b00aaf748dcc4f67727
Summary:
Adds a new method, `files_attrs`, to the `EdenApi` trait, which allows the caller to specify per-key attributes.
This method is intended to be temporary, and should later be unified with `files`.
Implement `files_attrs` in `FakeEdenApi`, and implement a placeholder method in EagerRepo.
Reviewed By: DurhamG
Differential Revision: D29635233
fbshipit-source-id: d0773927939527d799918139e4abba5ea3b5efca
Summary: Update `FileStore` to use new `indexedlogauxstore` rather than JSON + `indexedlogdatastore`.
Reviewed By: DurhamG
Differential Revision: D29635152
fbshipit-source-id: 3f73b7f7ee1ebc4aa1a0e804973d98d342cbc6ba
Summary: Add utility methods to ContentHash for use in computing aux data
Reviewed By: DurhamG
Differential Revision: D29659855
fbshipit-source-id: fb5c9749899147ea03dbb9e0e19b492c62bde2dd
Summary: Add a new indexedlog wrapper, `indexedlogauxstore`, for storing file aux data for scmstore.
Reviewed By: DurhamG
Differential Revision: D29597641
fbshipit-source-id: 34a9d9095ee580b4d210c82760691496358e0c6d
Summary:
Implement the `content` attribute.
Introduce a new `FileContent` type which stores the hg file blob and metadata, and modify `FileEntry` to allow constructing `FileEntry` with optional `FileContent` builder-style.
Reviewed By: DurhamG
Differential Revision: D29647203
fbshipit-source-id: b956c294d03dc81affc90d7274b2e430a3556e96
Summary:
Add support for optional file attributes to EdenApi, with `content` there as a placeholder.
Modifies the `FileRequest` type, adding a vec of `FileSpec`, which allows the client to specify desired attributes per-key. The existing `keys` field will be treated as a request for the content attribute and may be used in combination with the new per-key attributes.
Reviewed By: DurhamG
Differential Revision: D29634709
fbshipit-source-id: 6571837f87d1635e8529490e10dbe4ba054b7348
Summary: `parse_key` will be re-used in my incoming EdenApi aux data change.
Reviewed By: kulshrax
Differential Revision: D29601071
fbshipit-source-id: 2039c8478e717ff58af2030588dd31ec0b418b19
Summary:
We have a variety of enums in revisionstore that all serve the same purpose. With this change, I've consolidated them to a single type.
Not 100% sure if `StoreType` is the best name, in scmstore I named it `LocalStoreType` to be more clear - open to suggestions. I also decided to keep the variants named `Local` and `Shared` instead of adopting the `Local` and `Cache` terminology I used in scmstore - I'd rather not change that unless we decide to change the terminology everywhere to avoid confusion.
Reviewed By: kulshrax
Differential Revision: D29598025
fbshipit-source-id: 76d5a02230a8c1e5327cc5d90bbcae70049f251d
Summary:
Introduce a basic implementation of per-backend metrics collection for `FileStore`.
Currently, only indexedlog, lfs, and contentstore backends are instrumented, and only with basic metrics. Additional metrics (size, elapsed time, lfs pointer hits) and additional backends (aux, edenapi, lfs server) will be added in a later change.
Reviewed By: kulshrax
Differential Revision: D29552888
fbshipit-source-id: 54267f4de6f14db046cfae271b5c25a6bb494d7c
Summary:
Support exchange of mutation information during changesets uploads
Add new api for mutation information.
Add implementation for this new api.
Add client side support.
Reviewed By: markbt
Differential Revision: D29661255
fbshipit-source-id: 1d8cfa356599c215460aee49dd0c78b11af987b8
Summary:
API for uploading HgChangeset
This is the last part for uploding commit cloud commits via edenapi.
The structures are generic, could be reused for pull, etc
Reviewed By: markbt
Differential Revision: D29644799
fbshipit-source-id: 53347992cf399d99eaee4b5d2ad5f6ea30417022
Summary: This allows to use EDENSCM_LOG env variable to control trace logging from linked rust libraries
Reviewed By: fanzeyi
Differential Revision: D29641903
fbshipit-source-id: cc7395a2a5740c557e3166b786343ec0fefd6039
Summary:
Original commit changeset: 53d067bc2bb9
Same as https://www.internalfb.com/diff/D29586388 (0be0a68cca).
D29586388 (0be0a68cca) was disconnected from its parent and landed too early by mistake and reverted (because it wouldn't even compile without the parent commit).
This is just another version for the D29586388 (0be0a68cca) as it's not possible to reopen the original one.
Reviewed By: markbt
Differential Revision: D29598758
fbshipit-source-id: 96b6137366e196f9cd3c35c178eaa3d2fce1e071
Summary: when hg client is using x2pagentd it is not using TLS but plain HTTP. We shouldn't respect use-lfs-certs config option in this case.
Reviewed By: johansglock
Differential Revision: D29613166
fbshipit-source-id: 3a7c9c5add974dd927f4c76f1da2d5b8b67e864b
Summary: Log the key, metadata, and size of contentstore fallback hits in order to assist in debugging them.
Reviewed By: kulshrax
Differential Revision: D29552730
fbshipit-source-id: c10ed9dd50c48a28c2a256b9175e8555ea0862b2
Summary: There were bunch of warnings when compiling locally with debug_assertions
Differential Revision: D29594303
fbshipit-source-id: 7d257ff3d2450bfe8a089246b18511eb091ca361
Summary:
We had an issue in native checkout when update needed to remove a symlink and then create a directory with same files instead of symlink.
This used to fail, because update has a plan to write to new files, but the files has already 'existed' as part of a symlink, so unknown files check were failing.
This code makes sure that when listing untracked files we do not go inside symlinks, and treat audit errors from VFS as if file did not exist
Reviewed By: DurhamG
Differential Revision: D29567562
fbshipit-source-id: 1b6751cc00c3c628e2cab8c081540dba200209fa
Summary:
Add new client side API for upload trees.
Before uploading them, check what is already present on the server, similar as we check for filenodes.
I also added --force flag for the `hg cloud upload` command. It should be useful in general and useful for testing.
Reviewed By: markbt
Differential Revision: D29586388
fbshipit-source-id: 73c549f1a0d4328a64a133ab508fb4d253a4c33d
Summary:
upload filenodes (client side)
On the client side I implemented file upload and filenodes upload in the same API repo.edenapi.uploadfiles
This is because we should use the tokens from the file upload part to feed then into filenodes upload request.
Reviewed By: markbt
Differential Revision: D29549091
fbshipit-source-id: 436de187c8dce9a603c0c0a182e88b582a2d8001
Summary: There is no point to keep an empty buffer around.
Reviewed By: DurhamG
Differential Revision: D29565105
fbshipit-source-id: 1b8ea5e0158d89e119b01b1bbedd25dc280b44f3
Summary:
Reenables dynamicconfig loading with eden backingstore. Previously it
broke edenfs-windows-release, but we believe the
opensource/fbcode_builder/manifests/eden tweak has fixed it.
Reviewed By: xavierd
Differential Revision: D29561192
fbshipit-source-id: 775dd21d177f3baa09b0192e7d3f7231008c3766
Summary:
cpython-ext is not part of hg business APIs. It does not need to be consistent
with lower-case class name (and hg codebase is okay with non-lowercase class names,
just that most classes there are lower-case).
This resolves a rustc warning about the struct name.
Reviewed By: kulshrax
Differential Revision: D29526579
fbshipit-source-id: a4bc8e788d55c65aae9eaa87e2c684c2fded7ae2
Summary: The use case here is that we can export rust struct into python object, manipulate python object and then convert it back into rust struct
Reviewed By: quark-zju
Differential Revision: D29525281
fbshipit-source-id: 393c8a430a76c720c155432d732085892d56521a
Summary:
Previously Serde<T> required T to implement bunch of traits like Eq, Clone etc
This diff lifts this restriction and only implements those traits for Serde if T implements them, but does not require them
Reviewed By: quark-zju
Differential Revision: D29521112
fbshipit-source-id: e9387b08ab129fc1f2450f6c88fa7fe0f6451349
Summary:
We currently encode/decode data when passing it between pull_fast_forward_master and importpulldata and some other cases
pycell is alternative to that, allowing to just transfer pointer, instead of encoding data
Reviewed By: quark-zju
Differential Revision: D29474444
fbshipit-source-id: aabbbb670fde08645031c199afbdeab5cc9093d8
Summary: We'd like to eliminate directly writing serialized LFS pointers, but it turns out this is still necessary even when `remotefilelog.lfs=True` in some cases. With this change, we can enable writing serialized LFS pointers directly with a config.
Reviewed By: kulshrax
Differential Revision: D29496358
fbshipit-source-id: 66f53e0f0de52060920658a1781c32f27984ff3a
Summary:
Eliminate remaining uses of `contentstore` directly in the Python bindings.
Add repack functionality to `LegacyStore`.
Implement repack methods on `FileStore` by calling into contentstore fallback if possible, otherwise directly.
Reviewed By: kulshrax
Differential Revision: D29469901
fbshipit-source-id: 5cd134d25680bb55b4ddbe1f6d69e02af8942b16
Summary:
If the default is to pass certificates we should hard fail every single time they are not found instead silently making unauthenticated request. This will surface issues much quicker.
best_match_for can return `Ok(None)`.
Reviewed By: johansglock
Differential Revision: D29159532
fbshipit-source-id: ff28a627d91a9cf37258a97dc2c7f709ba8d00c2
Summary:
add 'upload/filenodes' request
This API must be called after file content has been uploaded. It requires a valid upload token for already uploaded file content.
The token can contain file content id of different types (canonical, sha1, sha256). It may or may not contain content size.
Reviewed By: StanislavGlebik
Differential Revision: D29197219
fbshipit-source-id: 3de31831ab06265675617a5c43cbd4be91f5cbe2
Summary:
Similar to D29404057 (cedddd1c8d), add a way to disable resolving IDs by setting
a limit using `EDENSCM_REMOTE_ID_THRESHOLD`.
Reviewed By: andll
Differential Revision: D29440143
fbshipit-source-id: 30409089493ae2cd5c189e37b0d4f88df9a6d8e8
Summary: Will be used by the next change.
Reviewed By: andll
Differential Revision: D29434722
fbshipit-source-id: 74dbec506fb0985379480815380118cd41058aec
Summary: This will be used by the next change.
Reviewed By: DurhamG
Differential Revision: D29428879
fbshipit-source-id: 69e0ffac12fb9c442488d59ea8faa0ea4b47a2c1
Summary: This makes it easier to figure out a traceback using an empty repo name.
Reviewed By: DurhamG
Differential Revision: D29428881
fbshipit-source-id: 95a09c691e3d921ad4f960a39002f71ec879d927
Summary:
This makes it easier to debug wrong reponame issues.
In theory those need to be checked and the config needs to be regenerated if
they are changed for correctness. Given that username and reponame are rarely
changed, I saved it for later.
Reviewed By: DurhamG
Differential Revision: D29428880
fbshipit-source-id: f996af6a7a1e329faaa8b0a53dac8621fa94dac8
Summary: Left over of D29404901 (410769c529).
Reviewed By: DurhamG
Differential Revision: D29429146
fbshipit-source-id: b37c89745d924efc28110d8b96e9b51162b6570b
Summary:
See also D29400532 (909411bb1c). It turns out that it might be more desirable to just mix
stdout and stderr streams in streampager. For example, having them mixed then
the graph log output can show what network fetches or calculations are done
before outputting the graph lines. This is also more consistent with the
vanilla terminal (no pager) behavior.
Reviewed By: markbt
Differential Revision: D29412531
fbshipit-source-id: c07f68b12498a7cee6152bbecbb58d5a7e64097a
Summary: Move the scmstore implementation from the `specialized` module to the root of the `scmstore` module.
Reviewed By: kulshrax
Differential Revision: D29405779
fbshipit-source-id: ae2ef9cc05337a0ff81f5ba5b7051792207fee82
Summary: `scmstore` is dead, long live `scmstore`.
Reviewed By: kulshrax
Differential Revision: D29405613
fbshipit-source-id: 3252a545f5b944d14c15b2a777b84a99a2d4c293
Summary: Update unit tests in `revisionstore::indexedlogdatastore` to use new scmstore instead of old scmstore.
Reviewed By: kulshrax
Differential Revision: D29405258
fbshipit-source-id: 3d2e8cd313dbe66a257433702402804f490bdf47
Summary:
Update unit tests in `revisionstore::edenapi::data` to use new scmstore. There's not really a wrapper to exercise anymore for edenapi specifically, so it's probably better to just make these `scmstore` unit tests instead of edenapi (or indexedlogdatastore as in the next change)-specific.
For ease of unit testing, make fetch_logger optional and introduce `empty` constructor.
Reviewed By: kulshrax
Differential Revision: D29397495
fbshipit-source-id: d7ef0df16cf83a2506606c55c78fcbfa684904d7
Summary:
The server is expected to provide head (of all segs), parents (of each seg),
roots (of all segs). We checked roots and parents but only check head in debug
build. Let's check head in release build too.
Reviewed By: andll
Differential Revision: D29405816
fbshipit-source-id: 1a97eb52a9a0d1d444ae5dabd1a01f0786be9fa9
Summary:
Found by xavierd. Recent `os_info` bump now detects CentOS as OracleLinux.
Workaround it to keep our repo functional.
Reviewed By: xavierd
Differential Revision: D29410415
fbshipit-source-id: 1bd8183f46e3c2265aef119e9f96d9d05a5dbae6
Summary: This will be used by the next change.
Reviewed By: andll
Differential Revision: D29400533
fbshipit-source-id: e6b90bedd8d8a6cf9452dfb5c5f14f9980e12f62
Summary:
More straightforward way of D29404055 (e6ea02372c). Return the non-lazy set directly from
Rust. This avoids some overheads.
Note: ignoring whitespace will make reviewing easier.
Reviewed By: andll
Differential Revision: D29404901
fbshipit-source-id: 02e4766256863fe3fe258bcb318473355cd1efe4
Summary: This was used to narrow down issues.
Reviewed By: andll
Differential Revision: D29404054
fbshipit-source-id: 3bfdac332d63bdb13f40d5cf23dacec242b46d52
Summary:
With lazy changelog, for every commit hash that is unknown to the repo, it
needs to be resolved remotely. For commit hash that is also unknown to the
remote server, it can be bad because we don't have cross-process negative
caching and will trigger the remote resolution over and over. Practically,
the solution is to avoid remote lookup if the "minor" correctness issue
is acceptable (ex. D29111710 (ac6c6cf3fa), D29114049 (880b5c3cd8)).
That has been tricky to debug - the remote fetching happens in Rust, and
cannot be easily inserted something like `import ipdb; ipdb.set_trace()`
like Python. I have been inserting sleeps and use gdb to understand the
call stack when writing the above 2 diffs.
This diff makes debugging easier by supporting "break point" setting by
environ variables. So one can do something like:
$ EDENSCM_LOG=dag::protocol=debug hg ...
DEBUG dag::protocol: resolve names [435c235d65ccc4f95595d74478a617450c96c2e] remotely
$ EDENSCM_DISABLE_REMOTE_RESOLVE=435c235d65cc4f95595d74478a617450c96c2ec hg ... --traceback --debugger
Reviewed By: andll
Differential Revision: D29404057
fbshipit-source-id: d8a631f279f32e2ee88f097796cdc85d8ca27b77
Summary:
Straightforward conversion of native checkout implementation from old scmstore API to new (non-async, batched) scmstore API.
We'll meet again someday soon, async.
Reviewed By: andll
Differential Revision: D29321512
fbshipit-source-id: 1e3e0d92c95730a5c2df610061f6faf5b1eb9068
Summary: The returned value now includes roots. Rename the function to clarify.
Reviewed By: kulshrax
Differential Revision: D29383072
fbshipit-source-id: 02a255ce20d9797f482f6fe1c716f2d79a12d4e0
Summary: There is a regression in 1.7.0 (which we're on at the moment) so we might as well update.
Reviewed By: zertosh, farnz
Differential Revision: D29358047
fbshipit-source-id: 226393d79c165455d27f7a09b14b40c6a30d96d3
Summary:
An alternative to D29363808 (e396cab669). The benefit is that parents_and_head is used by
both the client and the server. So we don't need to duplicate D29363808 (e396cab669) in
Mononoke code.
Reviewed By: andll
Differential Revision: D29365079
fbshipit-source-id: bca60ba2b3df477929d8e72b2363e5a0f744b35d
Summary:
Pull fast path uses `reload` which drops pending changes.
To avoid misuse, raise an error if pending changes are present.
Reviewed By: andll
Differential Revision: D29363799
fbshipit-source-id: 8f520d2c5553432abc452bc7b2b59d7af80e0a99
Summary:
The import pull data logic used low-level locking, persisting APIs, it does not
write cached idmap to disk. So we need to manually insert the idmap remote
lookup result to the actual local idmap explicitly.
This addressed an issue that verify_missing fails in the pull fast path.
Reviewed By: andll
Differential Revision: D29363813
fbshipit-source-id: 2749855a6c8c924bd1b772173de066d400f73764
Summary:
For a NameDag, `IdConvert` on `self.map` cannot resolve names remotely, but
`IdConvert` on `self` can. Use the latter. This is similar to D27547584 (af3c3b3fd0) where
some `self.map` are updated to `self`.
This addressed an issue found in the pull fast path test. Note there is another
issues to solve.
Reviewed By: andll
Differential Revision: D29363810
fbshipit-source-id: 28ba583ed14bbc5d52af81d4128d965f24eef011
Summary: The test pulls when the client has a lazy graph, and the server has a few merges.
Reviewed By: andll
Differential Revision: D29363806
fbshipit-source-id: 09bc3c4c3d21924f500ca86e8d86f58a15159169
Summary:
`fmt::Debug` for a NameDag is too verbose. Separate part of it so we can debug
print segments for a given (group, level). This will be used by upcoming
changes.
Reviewed By: andll
Differential Revision: D29363805
fbshipit-source-id: e1c6713be10b8b64fc7a42178117e724e0d691d0
Summary:
The client TestDag might have outdated server Dag as the remote protocol,
because it is a static "snapshot". Ensure the remote Dag is updated when
using the pull API.
This is an issue solved by tracking down issues in tests added in upcoming
diffs.
Reviewed By: andll
Differential Revision: D29363807
fbshipit-source-id: a560b2e91999873338604907a6d83cc7d2ff5c58
Summary: It will be used by the next change.
Reviewed By: andll
Differential Revision: D29363802
fbshipit-source-id: 842735ac05ea5fea4ea0c3625a68d06d27bc37d5
Summary:
It is useful when drawdag itself triggers remote fetches.
This was used but is not used after some refactoring. I think it might be useful
in the future so kept it.
Reviewed By: andll
Differential Revision: D29363803
fbshipit-source-id: fa178ac9783d1dc1b73525eeb8cd3d766cf46a0f
Summary: The test will be used to verify upcoming changes.
Reviewed By: andll
Differential Revision: D29363809
fbshipit-source-id: d34d13123914cfabb5c82dee3873b6e0c4979ee2
Summary: Make it easier to write more tests around pull.
Reviewed By: andll
Differential Revision: D29363804
fbshipit-source-id: 5b2cf8675343898fabc1d8845228e240e463edf8
Summary:
The roots data will be useful for the client to check if the pulled commits are
going to overlap with its existed DAG.
Reviewed By: andll
Differential Revision: D29363808
fbshipit-source-id: e09d924d65537f59fd4ea209b568265d07a80e46
Summary: Minor change to make the code a little bit more straightforward.
Reviewed By: andll
Differential Revision: D29363801
fbshipit-source-id: 2c4bd6ece07282f044622227a3c077cb31db6d17
Summary: Make the docstring a bit more consistent.
Reviewed By: andll
Differential Revision: D29363798
fbshipit-source-id: 1b4e2a7a1af4c4cffe3693e437a831bab1b43fd7
Summary: Verify we actually have pending keys to fetch before attempting a remote request in scmstore TreeStore.
Reviewed By: kulshrax
Differential Revision: D29345214
fbshipit-source-id: 328bdcbc41429e59de6ceb488533bafa97518fcc
Summary: Previously, it was not possible to interrupt `hg` during EdenAPI fetch operations. This made it impossible to interrupt long-running fetches, which is very frustating to users. This can be simply fixed by using `block_unless_interrupted` in place of `block_on`.
Reviewed By: quark-zju
Differential Revision: D29344670
fbshipit-source-id: 3b0d36dda28f5f7cc812a07981f295f8d0fbdd8a
Summary:
This is simple command mostly to be used by testing before we fully integrate with hg pull
This command does not perform discovery and requires from/to revision to be passed in cmd line
Reviewed By: quark-zju
Differential Revision: D29315647
fbshipit-source-id: 26d67031e566b7c99af1e2a5ab287f02b52f7db0
Summary: This is breaking the Windows release, reverting.
Reviewed By: fanzeyi
Differential Revision: D29339787
fbshipit-source-id: 22d8ff5db5619194e4597754dc37343cf0bc3286
Summary:
Introduce basic contentstore fallback tracking to help monitor the scmstore shim rollout.
This will be expanded to a general fetch metrics system for scmstore in a future change.
Reviewed By: kulshrax
Differential Revision: D29305839
fbshipit-source-id: c6cc3ea15a3bb7b90f4ec298febc911ec4e2af91
Summary:
Prevent `FileStore` from deadlocking when a write falls back to contentstore and attempts to write to the same indexedlog_local which is held lock for the batch.
Note: this shouldn't need to block release, we current expect writing raw LFS pointers to only happen with non-remotefilelog LFS.
Reviewed By: kulshrax
Differential Revision: D29299050
fbshipit-source-id: bf39f87b9956165a558f3a19960d3d055685db9a
Summary:
Introduce `LegacyStore` trait, which contains ContentStore methods not covered by other datastore traits.
Implement this trait for both contentstore and scmstore, and modify rust code which consumes `contentstore` directly to use `PyObject` and `LegacyStore` to abstract over both contentstore and scmstore instead.
Reviewed By: DurhamG
Differential Revision: D29043162
fbshipit-source-id: 26e10b23efc423265d47a8a13b25f223dbaef25c
Summary: Previously, we just fetched "best effort", and logged any encountered errors using `tracing`, leaving it up to the client to inspect errors if necessary. Python relies on catching these fetch errors as exceptions, though, so this change introduces some utility methods to help propagate them correctly.
Reviewed By: DurhamG
Differential Revision: D29211683
fbshipit-source-id: 5e9dee942c2b60e0f77a051624d7f393a811fc4e
Summary: My previous fix was actually incorrect, we now log actual remote requests, but join that with the logs from the contentstore fallback.
Reviewed By: DurhamG
Differential Revision: D29206878
fbshipit-source-id: d22e58792bf380c274e8086ce08aebe20dd9b848
Summary:
Previously, when fetching data using several concurrent requests, the EdenAPI client would wait for the headers for every request to finish coming in before starting to deserialize and yield entries from the bodies of any of the requests.
Normally, this isn't a huge deal since the response headers on all of the requests are usually roughly the same size, so they all finish downloading at roughly the same time when the requests are run concurrently. However, this does become an issue when `edenapi.maxrequests` is set. This option makes EdenAPI configure libcurl to queue outgoing connections once the configured limit is hit.
This means that although from EdenAPI's perspective all of the requests are running concurrently, they are not actually running in parallel. The result is that the EdenAPI client ends up waiting for all of the queued requests to be sent before yielding any data to the caller, which forces it to buffer all of the received data, resulting in massive memory consumption.
This diff fixes the problem by rearranging the structure of the Futures/Streams involved such that the client immediately begins yielding entries when they are received from any of the underlying transfers.
Reviewed By: quark-zju
Differential Revision: D29204196
fbshipit-source-id: b6b56bb7d60457de3c4046a07a5965749e9dd371
Summary:
When the `send_async` method is used to dispatch multiple concurrent requests, the method needs to return an `AsyncResponse` for each request. Since `AsyncResponse`'s constructor is itself `async` (it waits for all of the headers to be received), internally the method ends up with a collection of `AsyncResponse` futures.
Previously, in an attempt to simplify the API, the method would insert all of these futures into a `FuturesUnordered`, thereby conceptually returning a `Stream` of `AsyncResponses`. Unfortunately, this API ends up making it harder to consume the resulting `AsyncResponses` concurrently, as one might want to do when streaming lots of data over several concurrent requests.
This diff changes the API to just insert the `AsyncResponse` futures into a `Vec` to allow the caller to use them as desired. To maintain compatibility with the old behavior for the sake of this diff, the one current callsite has been updated to just dump the returned `Vec` into a `FuturesUnordered`. This will be changed later in the stack.
Reviewed By: quark-zju
Differential Revision: D29204195
fbshipit-source-id: ecee8cff430badd8213c2efef62fc68fbd91fde9
Summary: Nothing was using this metadata, and removing it simplifies the subsequent diffs in this stack.
Reviewed By: quark-zju
Differential Revision: D29147228
fbshipit-source-id: aa4828b710c3ef719f4d66adec5f66cd5b7d05d1
Summary:
This dep got updated in D29165283 (b82c5672fc) across a major version but the code depending
on it wasn't so now it's broken.
Reviewed By: mitrandir77
Differential Revision: D29229087
fbshipit-source-id: 5f2a14dd9f0447dd4578e8321991dfb3df32dcc2
Summary: Update versions for several of the crates we depend on.
Reviewed By: danobi
Differential Revision: D29165283
fbshipit-source-id: baaa9fa106b7dad000f93d2eefa95867ac46e5a1
Summary:
add an option to pass some metadata in the token
This will be used for content tokens, for example. We would like to guarantee that the specific content has been uploaded and it had the specific length. This will be used for hg filenodes upload.
Reviewed By: markbt
Differential Revision: D29136295
fbshipit-source-id: 2fbd3917ee0a55f43216351fdbc1a6686eb80176
Summary:
use `commitknown` edenapi api for checking the existing commits
it uses the same `lookup_commits` under the hood but a bit shorter to use
we won't need the tokens for existing changesets, so can use a simpler api
also, make `lookupfilenodes` function a bit shorter
Reviewed By: markbt
Differential Revision: D29134677
fbshipit-source-id: 257624d64480102c34761560b2bd768049cbfa83
Summary: They will be reused in import_pull_data
Reviewed By: quark-zju
Differential Revision: D29147950
fbshipit-source-id: 192bf33c30067f43c4fcaaf3054741b39efb4e25
Summary: This is an interface for importing pull data into dag
Reviewed By: quark-zju
Differential Revision: D29142979
fbshipit-source-id: b40b94403a044c0b74d1574528aa374ec309a0cf
Summary: This will be used to import pull data into segmented changelog
Reviewed By: quark-zju
Differential Revision: D29142981
fbshipit-source-id: 2d19a035ee0b6cefef8fc0547a5dfb79f284a1de
Summary:
The only real change here is: https://github.com/BurntSushi/ripgrep/pull/1756
This is a patch release but fixes a very glaring bug that others have
depended on. This diff fixes the uses to match the old behavior.
Although it's billed as a "fix", it's actually a huge perf improvement
for Linttool, which uses predominantly recursive suffix globs. The fact
that we don't have to compile ~5,000 regexps at Linttool startup anymore
makes such a huge difference that I am going to do write up soon.
Reviewed By: ndmitchell
Differential Revision: D29085977
fbshipit-source-id: 304470e5fa8cb986738aa0d9dd941641684a9194
Summary:
For a while we had two methods in async runtime: block_on_future and block_on_exclusive, due to historic reasons
Recently those methods were calling same code, and now it is time to replace both of them and rename to block_on
Reviewed By: quark-zju
Differential Revision: D29121107
fbshipit-source-id: 5faa76ae181e491b55d799c23c9de1b4e80298f3
Summary:
The added command gives access to read (execute) dynamicconfig without
using an on-disk repo.
It can be used by the clone script to stage rollout lazy changelog, or just to
verify dynamic config changes without using a repo.
Reviewed By: DurhamG
Differential Revision: D29123072
fbshipit-source-id: e8856d816a636fa860bfcc9694306a4a37552523
Summary:
implement uploading file content via Eden API
* in this diff I aim to upload file content for the given set of filenodes
* also, the code would check with Mononoke using the lookup request what is already there and skip those
* also, this diff introduces calculation of blake2 hash (called ContentId) for file contents (we would probably need to store/cache those and the mapping from hg filenode id to the canonical Mononoke content_id)
* for every uploaded content EdenApi returns a token that we would also need to store later
Reviewed By: markbt
Differential Revision: D29063229
fbshipit-source-id: 739a44bc3ff904cb04a39514ba5efd01c80ba6d0
Summary: This will be used in eager repo integration tests
Reviewed By: quark-zju
Differential Revision: D29113218
fbshipit-source-id: a24232bd6c19010d8ac90d1305f57f1094b06323
Summary:
The mutationstore might contain lots of names (commit hashes) that are unknown
to the repo. When using a lazy changelog, by default, those names will be
looked up remotely, and most of the time the server just returns "name is outside
the graph".
We don't have negative cache to speed it up, because cache invalidation is
tricky - every lazy pull would in theory invalidate them.
To make things faster, let's just skip names in the mutationstore that are unknown
_locally_ without asking the server.
In theory this might affect correctness. Practically, this should only affect "landed as" markers,
because all drafts should be non-lazy. If the "landed as" correctness is an issue, we can fix
forward "landed as" later (ex. by writing down the public commit hash explicitly in
debugmarklanded).
Reviewed By: andll
Differential Revision: D29111710
fbshipit-source-id: 2c1b16a8140ca4f7195bf1206ffe1db4750185b9
Summary: This allows BTreeMap to be used as a provide of config for testing.
Reviewed By: andll
Differential Revision: D28019023
fbshipit-source-id: 94a1585139d6c6612fa163f2ff1aecc909db3a72
Summary: Bumping the crossbeam version so we can use `recv_deadline`. This also necessitates updating the lsp and lsp-types crates.
Reviewed By: alunyov, dtolnay
Differential Revision: D29056473
fbshipit-source-id: 9434e9e0895d82482f4c70afa01a2f77702b965f
Summary:
This functionality is exercised in `test-lfs-copytracing.t` and a few other tests, though I'm not sure if it's necessary outside of that. I've added a `TODO` to investigate and try to eliminate such cases.
Because we'd rather not support this, I implemented it on top of ContentStore - it looks like we might have some lingering datapack-related tests where I think this comes up, too.
Reviewed By: DurhamG
Differential Revision: D29056647
fbshipit-source-id: e6bd5ec31dde06d8c509665f738cfe19270ddc76
Summary: Temporarily implement `upload` on `FileStore` by forwarding to the existing `upload` implementation using `LfsRemote`. Eventually we'll want to re-implement this entirely inside `FileStore`.
Reviewed By: DurhamG
Differential Revision: D29045264
fbshipit-source-id: 8ebbaead4c1032949e5bdd1dcf9a733d0086a77c
Summary: Add a `get_shared_mutable` method to `TreeStore` which behaves like `ContentStore::get_shared_mutable`, returning a store which will read/write only to the local cache stores (aka shared stores).
Reviewed By: DurhamG
Differential Revision: D29042716
fbshipit-source-id: f2236ff8b47ef213b2ffc61501ca301da02dc492
Summary: Add a `get_shared_mutable` method to `FileStore` which behaves like `ContentStore::get_shared_mutable`, returning a store which will read/write only to the local cache stores (aka shared stores).
Reviewed By: DurhamG
Differential Revision: D29042485
fbshipit-source-id: e28af0bafac5eba87523b0ef522f32355106b467
Summary: Add `FetchLogger` to `FileStore` to track remote fetches which match `remotefilelog.undesiredfileregex`.
Reviewed By: DurhamG
Differential Revision: D29042243
fbshipit-source-id: 08ec69d979d66ef93e8bbe8659171a412ade7e22
Summary: `ReportingRemoteDataStore` wraps a store and records all fetches which match `remotefilelog.undesiredfileregex`. Since scmstore doesn't use a hierarchical store composition, this change extracts out the matching/reporting logic into `FetchLogger` so that it can be re-used by both `ReportingRemoteDataStore` and scmstore.
Reviewed By: DurhamG
Differential Revision: D29041488
fbshipit-source-id: db0c4b545886ffebcf2a2841a506d301b2f2f230
Summary: Straightforward update of `debugscmstore` from old scmstore to new scmstore. I'll want to improve this command a bit more in the future, but this at least enables you to easily test arbitrary fetches.
Reviewed By: DurhamG
Differential Revision: D29047527
fbshipit-source-id: 4e10cb88cba4b572d3e413640ca3d800940d675d
Summary:
Our log deletion is a two step process. First it deletes the meta file,
then it deletes the log. There was a bug where if the meta file deletion
succeeded but the log deletion failed (often because Eden has the log open) then
future attempts to delete the log will fail because it fails to delete the meta
file.
Reviewed By: quark-zju
Differential Revision: D29035913
fbshipit-source-id: 33ab9a4e46b285819c4bf14031426983d2b4342f
Summary:
Since all the errors are eaten, we don't have good insight into the
rotatelog cleanup process. Windows users were seeing large growth in their
hgcache and this logging would've been useful, so let's add it.
Reviewed By: quark-zju
Differential Revision: D29035914
fbshipit-source-id: 54f0f05aa8bed1911b6b95438cd2d3ccc45e8ec9
Summary: This method will be used to get all Ids that needs to be included into CloneData::id_map during fast forward pull.
Reviewed By: quark-zju
Differential Revision: D29045538
fbshipit-source-id: f6eb04f537aa5365f7588391ea4c60b3dc010a53
Summary: revert the zstd crates back to previous version
Reviewed By: johansglock
Differential Revision: D29038514
fbshipit-source-id: 3cbc31203052034bca428441d5514557311b86ae
Summary:
implement `known` method based on the existing `lookup` API that has been already implemented
this might be useful for eden api pull project / wireproto deprecation, as the pull algorithm queries the server for whether commits are known by the server
Differential Revision: D29030429
fbshipit-source-id: 4d78c8b1e7983d89de917951d9765b1bd1d90fb0
Summary: This will be used in fastpath pull
Reviewed By: quark-zju
Differential Revision: D29010304
fbshipit-source-id: 9a7464fbc42544f8c5cc52f6fa25db5f7b5d6bd9
Summary: idset_to_flat_segments allows to return intersection between flat segments and span set
Reviewed By: quark-zju
Differential Revision: D28980521
fbshipit-source-id: da90c66a021a1e8bb95edbbc2e06cd2a850d7dcd
Summary:
This diff introduces Subspan trait for objects that contains spans and allows to take a 'subspan' of an object
Implementing Subspan trait allows to intersect arbitrary objects and spans
Reviewed By: quark-zju
Differential Revision: D28980523
fbshipit-source-id: 29a2e58ec3c79a838db751d8a37227b4cdaaa372
Summary: Update to latest version. This includes a patch to async-compression crate from [my PR updating it](https://github.com/Nemo157/async-compression/pull/125), I will remove once the crate is released.
Reviewed By: mitrandir77
Differential Revision: D28897019
fbshipit-source-id: 07c72f2880e7f8b85097837d084178c6625e77be
