Summary:
This is helpful, when we have raw unbundle bytes and a server path and just
want to send these bytes server's way.
Very similar to `sendunbundlereplay`, but does not do anything additional,
and reads from stdin.
Reviewed By: markbt
Differential Revision: D21527243
fbshipit-source-id: 97726cb40a32c7e44f47e0f56d8c8eabc4faf209
Summary:
As a developpers is working on large blobs and iterating on them, the local LFS
store will be growing significantly over time, and that growth is unfortunately
unbounded and will never be cleaned up. Thankfully, one the guarantee that the
server is making is that an uploaded LFS blob will never be removed[0]. By using
this property, we can simply move blobs from the local store to the shared
store after uploading the blob is complete.
[0]: As long as it is not censored.
Reviewed By: DurhamG
Differential Revision: D21134191
fbshipit-source-id: ca43ddeb2322a953aca023b49589baa0237bbbc5
Summary:
In order to build a StringPiece from a C-string, the length of the that
string needs to be known, and a constexp strlen is performed on it. That
strlen is however a recursive one, causing the stack to blow up on big file.
Interestingly enough, this also means that binary files potentially had a
wrong sha1 being computed, potentially causing `hg status` to report some
files as being modified, while they aren't. By switching to using a vector
instead of a string, the intent should of this should be more obvious.
Reviewed By: simpkins
Differential Revision: D21551331
fbshipit-source-id: 2dc1f08d96f49d310593e0e934a03215be2b5cbb
Summary:
If a push or pull operation doesn't involve any changesets for which mutation
information might be relevant, don't spend any time querying the database, and
instead exit early.
Reviewed By: krallin
Differential Revision: D21549937
fbshipit-source-id: a6f992e621456b826acd1bddde3591e751d23b31
Summary:
MySQL returns an error for a query of the form `WHERE value IN ()`. Avoid
these by checking that collections are not empty before making the query.
Reviewed By: krallin
Differential Revision: D21549690
fbshipit-source-id: 1507d36e81f7a743d2a1efb046e52a5479633ab9
Summary:
The `test-infinitepush-mutation.t` test covers the new mutation database, so
add it to the mysql tests.
Reviewed By: krallin
Differential Revision: D21548966
fbshipit-source-id: 0dc1f90129fa61fb6db1c1b5a747efa3d20041f5
Summary:
The `support_bundle2_listkeys` flag controls at runtime whether we support
`listkeys` in bundle2. Since this was added before tunables were available,
it uses a value in the mutable counters SQL store.
We could migrate this to tunables, but in practice we have never disabled it,
so let's just make it the default.
Reviewed By: krallin
Differential Revision: D21546246
fbshipit-source-id: 066a375693757ea841ecf0fddb0cc91dc144fd6f
Summary:
When the client pulls draft commits, include mutation information in the bundle
response.
Reviewed By: farnz
Differential Revision: D20871339
fbshipit-source-id: a89a50426fbd8f9ec08bbe43f16fd0e4e3424e0b
Summary:
Advertise support for `b2x:infinitepushmutation`. When the client sends us
mutation information, store it in the mutation store.
Reviewed By: mitrandir77
Differential Revision: D20871340
fbshipit-source-id: ab0b3a20f43a7d97b3c51dcc10035bf7115579af
Summary: Add the mutation store to blobrepo.
Reviewed By: krallin
Differential Revision: D20871336
fbshipit-source-id: 777cba6c2bdcfb16b711dbad61fc6d0d2f337117
Summary: What it says in the title
Reviewed By: StanislavGlebik
Differential Revision: D21549635
fbshipit-source-id: 75939ebbfb317a9beaa9acd1fc1a7c6f41b0f88f
Summary: The compiler is warning about it.
Reviewed By: singhsrb
Differential Revision: D21550266
fbshipit-source-id: 4e66b0dda0e443ed63aeccd888d38a8fcb5e4066
Summary: Expose the API that returns a real graph.
Reviewed By: DurhamG
Differential Revision: D21486520
fbshipit-source-id: 4ebdb4011df8971c54930173c4e77503cd2dac47
Summary:
Part of the mutation graph (excluding split and fold) can fit in the DAG
abstraction. Add a method to do that. This allows cross-dag calculations
like:
changelogdag = ... # suppose available by segmented changelog
# mutdag and changelogdag are independent (might have different nodes),
# with full DAG operations on either of them.
mutdag = mutation.getdag(...)
mutdag.heads(mutdag.descendants([node])) & changelogdag.descendants([node2]) # now possible
Comparing to the current situation, this has some advantages:
- No need to couple the "visibility", "filtered node" logic to the mutation
layer. The unknown nodes can be filtered out naturally by a set "&"
operation.
- DAG operations like heads, roots can be performed on mutdag when it's
previously impossible. We also get operations like visualization for free.
There are some limitations, though:
- The DAG cannot represent non 1:1 modifications (fold, split) losslessly.
Those relationships are simply ignored for now.
- The MemNameDag is not lazy. Reading a long chain of amends might be slow.
For most normal use-cases it is probably okay. If it becomes an issue we
can seek for other solutions, for example, store part of mutationstore
directly in a DAG format on disk, or have fast paths to bypass long
predecessor chain calculation.
Reviewed By: DurhamG
Differential Revision: D21486521
fbshipit-source-id: 03624c8e9803eb1852b3034b8f245555ec582e85
Summary:
subcommand_single calls `derived_data_utils.regenerate(vec![cs_id])` with the
intention that derived data for this commit will be regenerated. However
previously it didn't work because DerivedDataUtils::derive() was ignoring
regenerate parameter. This diff fixes it.
Reviewed By: krallin
Differential Revision: D21527344
fbshipit-source-id: 56d93135071a7f3789262b7a9d9ad84a0896c895
Summary:
This allows us to replace the pyre2 C++ bindings so the fast regex engine can
work with Python 3, and simplify our build steps.
Reviewed By: DurhamG
Differential Revision: D20973179
fbshipit-source-id: e123ac18954991f2c701526108f5c2ecd2b31a3b
Summary: Add a `/history` endpoint that serves EdenAPI history data. Like the other endpoints, this one currently buffers the response in memory, and will be modified to return a streaming response in a later diff.
Reviewed By: krallin
Differential Revision: D21489463
fbshipit-source-id: 259d2d1b7d700251fe902f1ac741545e5895404a
Summary: Add the ability to parse EdenAPI history responses to `data_util`.
Reviewed By: quark-zju
Differential Revision: D21489228
fbshipit-source-id: 42dda64273673431a6f3e4d7bd430689c76c387f
Summary: Factor out logic that will be common to many handlers into new functions in the `util` module.
Reviewed By: krallin
Differential Revision: D21489469
fbshipit-source-id: 9aff4e5182748ab0a0bedd6038852692b8e721a7
Summary: Break up the EdenAPI server integration tests to prevent the test from getting too long.
Reviewed By: krallin
Differential Revision: D21464056
fbshipit-source-id: 076aaf8717547fe9188f40c078d577961c02325d
Summary: Add an endpoint that serves trees. Uses the same underlying logic as the files endpoint, and returns the requested nodes in a CBOR DataResponse.
Reviewed By: krallin
Differential Revision: D21412987
fbshipit-source-id: a9bcc169644a5889c3118a3207130228a5246b2f
Summary: Change `make_req` to take a JSON array as input when constructing `DataRequest`s instead of a JSON object. This is more correct because DataRequests can include multiple `Key`s with the same path; this cannot be represented as an object since an object is effectively a hash map wherein we would have duplicate keys.
Reviewed By: quark-zju
Differential Revision: D21412989
fbshipit-source-id: 07a092a15372d86f3198bea2aa07b973b1a8449d
Summary: EdenAPI data responses can contain data for either files or trees. As such, the implementation of both the files and trees endpoints is almost identical. To allow the logic to be shared between both, this diff makes the handler code generic.
Reviewed By: krallin
Differential Revision: D21412986
fbshipit-source-id: 89501915b0401214b373ed1db2eb09e59de2e5b7
Summary: In order to allow writing code that is generic over files and trees, move the functionality common between the two to a separate trait. This will allow for a significant amount of code sharing in the EdenAPI server. (This diff does not introduce any new functionality; it's mostly just moving existing code into the new traits.)
Reviewed By: krallin
Differential Revision: D21412988
fbshipit-source-id: 31b55904f62ccb965b0f9425de875fc069e10b5a
Summary:
Add an endpoint that serves Mercurial file data.
The data for all files involved is fetched concurrently from Mononoke's backend but in this initial version the results are first buffered in memory before the response is returned; I plan to change this to stream the results in a later diff.
For now this version demonstrates the basic functionality as well as things like ACL enforcement (a valid client identity header with appropriate access permissions must be present for requests to succeed).
Reviewed By: krallin
Differential Revision: D21330777
fbshipit-source-id: c02a70dff1f646d02d75b9fc50c19e79ad2823e6
Summary: Nothing should be running with this version, so this check is not needed anymore!
Reviewed By: chadaustin, wez
Differential Revision: D21513269
fbshipit-source-id: 698c9d84e87461cd0484ef997f44dc9169e87a25
Summary:
While not all stats are supported on Windows, most of them can, let's enable
them. This is a first step in unifying the mount path for Windows and Unices.
Reviewed By: chadaustin
Differential Revision: D21514732
fbshipit-source-id: 00fd89b7499f8ae18c089f98bc18cd43f68538fa
Summary:
Write commit cloud sync and clienttelemetry blackbox logs to tracing data.
Note: since metalog can answer head changes, I didn't add head changes of
commit cloud sync to the tracing data.
Reviewed By: DurhamG
Differential Revision: D19797700
fbshipit-source-id: b89924a7aa5e6027cad5c8138e8988f6b0ea4b2a
Summary: This allows us to query tracing data for watchman commands.
Reviewed By: DurhamG
Differential Revision: D19797711
fbshipit-source-id: 4dfd50fff820da70888faa0fe8f53af25f205137
Summary:
The repo.pull API updates remote bookmarks on its own. Therefore do not even
ask the server to listkey (all) bookmarks.
This also removes the need of listkeys(bookmarks) for the new clone API.
Reviewed By: DurhamG
Differential Revision: D21011393
fbshipit-source-id: b10bdbc82563c32626bdcb2632170fd56819e904
Summary:
The core `clone.streamclone` is the new clean way to do a streaming clone with
selectivepull. Detect the use of it and skip remotenames' own exclone logic.
Reviewed By: DurhamG
Differential Revision: D21011396
fbshipit-source-id: 50fdbf4c2761a96c50e23f21a87ef636fac74afb
Summary:
The current `clone --shallow` command has some issues:
- It fetches *all* remote bookmarks, since selectivepull does not work with
streamclone, then remove most remote bookmarks in a second transaction.
- It goes through remotenames, which is racy, and D20703268 does not fix the
clone case. Possible cause of T65349853.
- Too many wrappers (ex. in remotefilelog, remotenames, fastdiscovery) wtih
many configurations (ex. narrow-heads on/off) makes it hard to reason about.
Instead of bandaidding the clone function, this diff adds a new clone implementation
that aims to solve the issues:
- Use streamclone, but do not pull all remote names.
- Pull selectivepull names explicitly with a working "discovery" strategy
(repo heads should be non-empty with narrow-heads on or off).
- Do clone in one transaction. Outside world won't see an incomplete state.
- Use `repo.pull` API, which is not subject to race conditions.
- Eventually, this might be the only supported "clone" after Mononoke becoming
the single source of truth.
Note: the code path still goes through bookmarks.py and remotenames.py.
They will be cleaned up in upcoming diffs.
Reviewed By: DurhamG
Differential Revision: D21011401
fbshipit-source-id: d8751ac9bd643e9661e58c87b683be285f0dc925
Summary:
In the past we hide the revlog headrevs API with the idea that calculating
heads in the DAG is not going to scale, and heads should be based on references
(remotenames, visible heads). Practically calculating heads in the DAG based
on segmented changelog is not going to be painfully slow so we probably can
afford it.
Therefore let's just re-expose the DAG-based heads API as rawheads. The only
user of it is in dagutil.py.
This will be used in the next diff where streamclone first gets the revlog
changelog copied without remote bookmarks. Then it needs to do a pull
which requires the heads information.
Reviewed By: DurhamG
Differential Revision: D21296530
fbshipit-source-id: a81a61e3b58c921a3390fda8f716bd7ae0e55ed1
Summary:
Move the logic to write repo hgrc ([paths]) and set [paths] config options
earlier, so other logic can use the [paths] config.
Some tests are changed because remotenames can now write bookmarks in more
cases.
Reviewed By: DurhamG
Differential Revision: D21011397
fbshipit-source-id: 4b921a02c20daeef31d44a03264a89b975303aa5
Summary:
"[paths] being empty" will no longer be a way to test initial clone, use
transaction name instead.
Reviewed By: DurhamG
Differential Revision: D21011395
fbshipit-source-id: e257fe8eb2efd45ac52fad7c74363151b0a8c417
Summary: This will be used by hg-git to test initial clone.
Reviewed By: DurhamG
Differential Revision: D21011400
fbshipit-source-id: 11a1a41631830273a6407e419ebe5ff21964e7de
Summary: It is not used and makes the already complicated clone logic more complicated.
Reviewed By: DurhamG
Differential Revision: D21011394
fbshipit-source-id: 3620f7372a9f3cefc60618052c768c6c2cbe04f9
Summary:
They are not used. Remove it to make it a bit easier to move stream_out_shallow
to core.
Note: this does not remove all include/excludepatterns yet.
Reviewed By: DurhamG
Differential Revision: D21011403
fbshipit-source-id: f6d27a3e2472f6c69f95a958ac99f75a8b8f8b74
Summary: It will be used in the next change.
Reviewed By: DurhamG
Differential Revision: D21011399
fbshipit-source-id: 6bdffc79af0474e42562686109417882a8cb2cd6
Summary:
While this isn't the right fix, this is what shipped in our packages, for the
sake of being able to reproduce the package, let's land this as it is. A
future change will remove this ifdef.
Below is pkaush original description:
In Eden Windows we treat all the files as regular files and don't have a
concept of symlinks and executable files. Fixing the TreeEntryType::getType()
to return REGULAR_FILE for executable file and symlink.
Reviewed By: wez
Differential Revision: D20481051
fbshipit-source-id: 0b0c4d7aea28134383ef45aeafc02930b420286b
Summary:
Right now, we debug-print the root cause and pretty-print everything else. This
is pretty bad because the root cause is usually the one thing we would want to
pretty print so we can add instructions there (such as "your hooks failed, fix
it").
This fixes this so we stop pretty-printing the root cause, but also debug print
the whole error, which gives us more developer-friendly context and is easier
for automation to match on.
This is actually in common/rust ... but we're the only people using it AFAICT.
Reviewed By: StanislavGlebik
Differential Revision: D21522518
fbshipit-source-id: 10158811574b56024e14852229e4541da19d5609
Summary:
Add the `hg cloud hide` command. This allows removal of commits, bookmarks and
remote bookmarks from a cloud workspace, even when the items are omitted
locally.
Reviewed By: DurhamG, quark-zju
Differential Revision: D21409384
fbshipit-source-id: 24b64c207c78f9b0258e9cf6a578db7b14c84901
Summary:
Limits on concurrent calls are a bit hard to reason about, and it's not super
obvious what a good limit when all our underlying limits are expressed in QPS
(and when our data sets don't have peak concurrency - instead they have
completion time + # blob accesses).
Considering our past experience with ThrottledBlob has been quite positive
overall, I'd like to just use the same approach in ContextConcurrencyBlobstore.
To be safe, I've also updated this to be driven by tunables, which make it
easier to rollout and rollback.
Note that I removed `Debug` on `CoreContext` as part of this because it wasn't
used anywhere. We can bring back a meaningful implementation of `Debug` there
in the future if we want to. That triggered some warnings about unused fields,
which for now I just silenced.
Reviewed By: farnz
Differential Revision: D21449405
fbshipit-source-id: 5ca843694607888653a75067a4396b36e572f070
Summary:
The motivation for making this function async is that it needs to spawn things,
so it should only ever execute while polled by an executor. If we don't do
this, then it can panic if there is no executor, which is annoying.
I've been wanting to do this for a while but hadn't done it because it required
refactoring a lot of things (see the rest of this stack). But, now, it's done.
Reviewed By: mitrandir77
Differential Revision: D21427348
fbshipit-source-id: bad077b90bcf893f38b90e5c470538d2781c51e9
Summary:
This updates our blobrepo factory code to async / await. The underlying
motivation is to make this easier to modify. I've ran into this a few times
now, and I'm sure others have to, so I think it's time.
In doing so, I've simplified the code a little bit to stop passing futures
around when values will do. This makes the code a bit more sequential, but
considering none of those futures were eager in any way, it shouldn't really
make any difference.
Reviewed By: markbt
Differential Revision: D21427290
fbshipit-source-id: e70500b6421a95895247109cec75ca7fde317169
