Summary:
This makes it easier to further migrate to `Text` interface.
Dependent crate (`auth`) is updated.
Reviewed By: markbt
Differential Revision: D20432941
fbshipit-source-id: 1dc29d52c9b17ce14676ef0555470c6d36a09c2b
Summary:
Text is a reference-counted shared String.
It's similar to Bytes but works for utf-8 strings.
The motivation is to replace configparser's use of Bytes to Text.
Reviewed By: markbt
Differential Revision: D20432940
fbshipit-source-id: ef990255d269e60d433c6520819f60ccdcbe488f
Summary: This makes it possible to implement "Text". See the next diff.
Reviewed By: markbt
Differential Revision: D20432943
fbshipit-source-id: 94b3810ab205c260d33f57bd637e4accc3ee871d
Summary:
This makes the API easier to use.
Practically this makes it easier for configparser to migrate to minibytes.
Reviewed By: markbt
Differential Revision: D20432942
fbshipit-source-id: ad08eb118d2216054dc24c86b0b129ae82b9d17c
Summary:
Next OpenNSA release with increased stack size (needed for FBOSS) is available.
Start linking with it.
Differential Revision: D20062997
fbshipit-source-id: 9938270c322087ac3990861aa1ddd3b9ea1148ac
Summary: small cleanup - will help remove clone in the later diff
Reviewed By: farnz
Differential Revision: D20441266
fbshipit-source-id: ee15c7bdbfe2f42ccd1d10abbbe0d43e89547df5
Summary:
let's return the generation numbers as part of commit info - they are cheap to
obtain.
Reviewed By: StanislavGlebik
Differential Revision: D20426744
fbshipit-source-id: 50c7017c55aeba04fb9059e2c1db19f2fb0a6e5e
Summary:
Let's return those to the caller. The changeset is already fetched at this
point so this should come at no extra cost.
Reviewed By: markbt
Differential Revision: D20426745
fbshipit-source-id: 7e28f5fd44efdb502b8e37a7afac3ea9113e2c5e
Summary: I'm about to introduce one more usecase of it so let's rename it first.
Reviewed By: farnz
Differential Revision: D20393776
fbshipit-source-id: d74146fa212cdc4989a18c2cbd28307f58994759
Summary:
Currently, x-repo commit validator runs full working copy comparison every time. This is slow, as it requires fetching of full manifests for the commit versions in both a small and a large repo. For those of leafs, which are different, filenodes are also fetched, so that `ContentId`s can be compared. It's slow. Because it's slow, we skip the majority of commits and validate only 1 in ~40 `bookmarks_update_log` entries. Obviously, this is not ideal.
To address this, this diff introduces a new way to run validation: based on comparing full manifest diffs. For each entry of the `bookmarks_update_log` of the large repo, we:
- unfold it into a list of commits it introduces into the repository
- fetch each commit
- see how it rewrites to small repos
- compute manifest differences between the commit and it's parent in each involved repo
- compare those differences
- check that topological order relationship remains sane (i.e. if `a` is a p1 of `b` in a small repo, than `a'` must be a p1 of `b'` in a large repo, where `a'` and `b'` are remappings of `a` and `b`)
In addition, this diff adds some integration tests and gets rid of the skipping logic.
Reviewed By: StanislavGlebik
Differential Revision: D20007320
fbshipit-source-id: 6e4647e9945e1da40f54b7f5ed79651927b7b833
Summary:
Replace ownership moves with borrows here and there, add fetching of `FileType`
as well.
Reviewed By: krallin
Differential Revision: D20387564
fbshipit-source-id: 378511402277d744ddfbb68e01a3f5d1707f6d08
Summary:
Even simplest test took ~30 seconds to finish. Turned out most of the time is
sepnt in creating sqlite shards, and the reason we haven't noticed it before is
because we haven't use sqlblob in integration tests until recently (until
February 24 to be precise - D20001261).
We don't need 100 sqlite shards, let's decrease it downto 5.
Reviewed By: HarveyHunt
Differential Revision: D20438107
fbshipit-source-id: e71fd4cabf908e3d92b446fc518a0e5dd64a00bb
Summary:
For users on slow connections, hg debugnetwork may not complete within 20s.
Reduce the amount of data we use for the download and upload tests, so that we
have a better chance for the tests completing within the 20s limit.
Reviewed By: farnz
Differential Revision: D20439271
fbshipit-source-id: b3cb0d7e96673c8052022c7606cca80db1788370
Summary: At current, there are some specific AOSP branches that are causing problems for the importer. It would be helpful to be able to run convert jobs that only run on a subset of branches for testing. This commit adds a configuration setting for a whitelist that allows us to specify which repo branches we want to convert, and we only convert those branches.
Reviewed By: tchebb
Differential Revision: D18534728
fbshipit-source-id: bf59605d9c6026e26914c4a7cbe9493bdee77ee5
Summary:
The input for getcommitdata is a list of HgChangesetIds. For every entry, the
endpoint retrieves the commit, formats it like the revlog then returns it back
to the client.
The format for the returned entries is:
```
Hash " " Length LF
RevlogContent LF
```
Looking for recommendations for how to structure the code better.
Looking for recommendations on implementation requirements:
metrics, throttling, veriftying hash for returned bytes.
Reviewed By: krallin
Differential Revision: D20376665
fbshipit-source-id: 5d9eb0d581fd2b352cf3ce44f4777ad45076c8f4
Summary:
The data portion of the changelog revlog from local checkouts is going away.
Instead of fetching data from disk, data will be fetched from the server.
For migration purposes we want the data from the server to match the data that
would have been available locally.
Note. I don't understand the difference between `RevlogChangeset` and
`HgChangesetContent`. I want to eventually call
`mercurial_revlog::changeset::serialize_cs` and this change reduces
the number of transformations required. The whole process is more direct and
thus easier to follow.
Reviewed By: StanislavGlebik
Differential Revision: D20373568
fbshipit-source-id: f2f4eccdf6a35b322e13c5a4b3fa18d5d289848e
Summary:
The diff only contains HgCommand signatures. No implementation yet.
The purpose of the getcommitdata command is to return the serialized contents
of a commit. Given a Mercurial Changelog Id, the endpoint returns the same
contents that the Revlog would return on a Mercurial server.
At this point I am looking for suggestions regarding the protocol and the
implementation. My assumption is that both request and response can be fully
kept in memory. I think that we may decide that the request is going to be
streamed to the client so the initial protocol allows for that.
Requirements:
Input: HgChangelogId
Output: Changelog entry contents
Protocol Summary:
```
Request: "getcommitdata" LF "nodes " Length LF Length*(HEXDIG|" ")
Response: *(40HEXDIG Length LF Length*(%x00-79) LF)
```
A bit of a silly protocol. Let me know what recommendations you have.
The Request is modelled after the "known" command. This allows for efficient
batching compared to a batch command model. It's a bit awkward that we don't
pass in the number of HgChangelogId entries that we have in the request but
that is the existing protocol.
For every HgChangelogId in the request the response will first have a line
with the HgChangelogId that was requested and the length of the contents.
The next line will contain the contents followed by line feed.
Reviewed By: krallin
Differential Revision: D20345367
fbshipit-source-id: 50dffff4f6c60396f564f2f1f519744ce730bf96
Summary:
The metalog message is just for display purpose so it does not have to be
byte-to-byte accurate. This solves potential crashes on Python 2 Windows.
File "c:\Tools\hg\python27.zip\edenscm\mercurial\transaction.py", line 587, in _writemetalog
" ".join(map(util.shellquote, pycompat.sysargv[1:])), int(util.timer())
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 1: invalid utf-8
The logic will become more "correct" when we migrate to Python 3.
Reported by: markbt
Reviewed By: markbt
Differential Revision: D20422747
fbshipit-source-id: 41123d132a1e545db77d7321099da611668174f4
Summary:
rust-crypto hasn't been updated since 2016, we are replacing it according to https://github.com/RustSec/advisory-db/blob/master/crates/rust-crypto/RUSTSEC-2016-0005.toml
Specifically, we replace crypto::Digest with digest::Digest, crypto::Sha1 with sha1::Sha1.
Reviewed By: krallin
Differential Revision: D20424524
fbshipit-source-id: aa5d268cb5d3c4b1b4e9f7c6f1f3c41d559f121a
Summary:
hg makes sure commit messages are utf-8. So there is no need to check it.
The check logic is also incorrect if the local encoding is not utf-8:
`ctx.description()` returns local-encoding-encoded bytes, which should not
be treated as utf-8 bytes.
Reviewed By: markbt
Differential Revision: D20424241
fbshipit-source-id: 05480e37335c3f2c7971c28c2e23c115dfc6fba7
Summary: This makes it possible to hide draft branches using `hg hide`.
Reviewed By: markbt
Differential Revision: D20403505
fbshipit-source-id: d316e02c24ce636fdc6593f95a5d974b1ba0fc85
Summary:
During cloud sync, the `_applycloudchanges` step will pull all the new commits
from commit cloud, and then record that it has updated to the new workspace
version in the cloud sync state.
If the transaction subsequently fails and gets rolled back, we will roll back
the pull (so the new commits vanish) but we will not roll back the recording of
having updated to the new cloud workspace version.
On the *next* cloud sync, commit cloud will think the user has deliberately
hidden all of those missing commits, and remove them from the cloud workspace.
Write the cloud sync state within the transaction, so if the transaction is
rolled back, we also roll back the recording of the new workspace version.
Reviewed By: quark-zju
Differential Revision: D20420240
fbshipit-source-id: f3d201f299e491cbff47fc6b703a4e1af4015b18
Summary:
Commit cloud sync works at the store granularity, so the status should be
stored in the store vfs. Move it there.
Improve the output of `hg cloud status` by logging more information about the workspace.
Reviewed By: quark-zju
Differential Revision: D20419223
fbshipit-source-id: 4f0d6b9aab55d2bbeaf89b489606f0bc25400de5
Summary:
Avoid boilerplate in unit tests by consistently using `fbinit::compat_test`.
This is a very mechanical change; most of the modified lines are due to indentation changes.
Reviewed By: krallin
Differential Revision: D20414564
fbshipit-source-id: 11c6bd702de28c8a21978f0345d2b0ff8a574e72
Summary:
Expose the "TreeSpans" type from tracing-collector so filtering can be done on
them.
Reviewed By: DurhamG
Differential Revision: D19797707
fbshipit-source-id: 56b63c8fb79865666ce923612dbd5a9cc512dce6
Summary:
Previously Rust str was serialized into bytes. To be Python 3 friendly, let's
serialize it into `str`.
Reviewed By: markbt
Differential Revision: D19797706
fbshipit-source-id: 388eb044dc7e25cdc438f0c3d6fa5a5740f22e3d
Summary:
Let's use functionality added in D20389226 so that we can generate
diffs even for large files.
I've contemplated between two approaches: either "silently" generate
placeholder diff for files that are over the limit or add a new option
where client can request these placeholders. I choose the latter for a few
reasons:
1) To me option #1 might cause surprises e.g. diffing a single large file
doesn't fail, but diffing two files whose size is (LIMIT / 2 + 1) will fail.
Option #2 let's the client be very explicit about what it needs - and it also
won't return a placeholder when the actual content is expected!
2) Option #2 makes the client think about what it wants to be returned,
and it seems in line with source control thrift API design in general i.e.
commit_file_diffs is not trivial to use by design to because force client to
think about edge cases. And it seems that forcing client to think about additional
important edge case (i.e. large file) makes sense.
3) The code change is simpler with option #2
I also thought about adding generate_placeholder_diff parameter to CommitFileDiffsParams,
but that makes integration with scs_helper.py harder.
Reviewed By: markbt
Differential Revision: D20417787
fbshipit-source-id: ab9b32fd7a4768043414ed7d8bf39e3c6f4eb53e
Summary:
This adds a test demonstrating that we can perform BFS fetching over SSH. The
test should demonstrate that the fetches are:
- Done in a BFS fashion (we don't fetch the entire trees before comparing them,
instead we do a fetch at each layer in the tree). In particular, note that
the cc tree, which is unchanged, doesn't get explored at all.
- Done using the right paths / filenodeids, and therefore the right linknodes
are located.
Reviewed By: farnz
Differential Revision: D20387124
fbshipit-source-id: b014812b0e6e85a5cdf6abefe3fe4f47b004461e
Summary:
This is convenient because it makes it possible to tell what is going on within
Mononoke from the outside (e.g. introspect perf counters).
I'll land this after D20387115.
Reviewed By: farnz
Differential Revision: D20387125
fbshipit-source-id: ee070c4d658a0ec8f232152fe8b34bd0b56e6888
Summary:
This diff adds client-side support for the new `designatednodes` mode of the `gettreepack` wireproto command (implemented in D20307442), which allows fetching an exact set of specified tree nodes. This enables the performance benefits of BFS tree fetching over SSH (whereas this sort of fetching was previously only possible over HTTPS via EdenAPI).
To allow us to smoothly control the rollout of this feature, it is gated behind the `treemanifest.ondemandfetch` config option. In theory, this should not be needed since the client will check that the server supports `designatednodes` before attempting to do on-demand fetching, but if users report issues it's nice to have a killswitch.
Reviewed By: quark-zju
Differential Revision: D20382825
fbshipit-source-id: f9aef3fdae08ae84a9720deead3549fe2fba0b9b
Summary:
Similarly to D20418610, let's keep track of request failure rates in ODS
counters so they can be health-cheked by SF.
Reviewed By: farnz
Differential Revision: D20418705
fbshipit-source-id: 1d281e06c59414a0610836106370592596b47e39
Summary:
Earlier, I added counters for successes and failures, but unfortunately
canaries do not allow for tracking formulas, so I also need the actual request
success rate logged.
More broadly speaking, it probably does make sense to have a very high level
"success rate" counter to query anyway. This adds it.
Reviewed By: farnz
Differential Revision: D20418610
fbshipit-source-id: 94f115f8201aef4f6498f4da98e170194186f0f8
Summary:
We cannot differentiate an empty list from a list containing the empty path only. This fixes that.
See D20309700 earlier in this stack for the server-side bits.
Reviewed By: quark-zju
Differential Revision: D20309699
fbshipit-source-id: 02efc747c3c5e2774f949ba0a6370169b55f3cf3
Summary:
We need to differentiate the empty directory from no directories. Adding a
trailing comma after each directory instead of separating them achieves that.
Reviewed By: StanislavGlebik
Differential Revision: D20309700
fbshipit-source-id: 387ec477560968392de0a9631d67ccb591bd3cab
Summary:
This will allow the hg client to do tree fetching like we do in the API Server,
but through the SSH protocol — i.e. by passing a series a manifest ids and
their paths, without recursion on the server side through gettreepack.
Reviewed By: StanislavGlebik
Differential Revision: D20307442
fbshipit-source-id: a6dca03622becdebf41b264381fdd5837a7d4292
Summary:
The goal of the stack is to support "rendering" diffs for large files in scs
server. Note that rendering is in quotes - we are fine with just showing a
placeholder like "Binary file ... differs". This is still better than the
current behaviour which just return an error.
In order to do that I suggest to tweak xdiff library to accept FileContentType
which can be either Normal(...) meaning that we have file content available, or
Omitted, which usually means the file is large and we don't even want to fetch it, and we
just want xdiff to generate a placeholder.
Reviewed By: markbt, krallin
Differential Revision: D20389226
fbshipit-source-id: 0b776d4f143e2ac657d664aa9911f6de8ccfea37
Summary:
In order to support gradual rollout of infinitepush for other backends (e.g. Mononoke), we need the ability to route read vs write requests separately. To achieve this, we need a separate path type: `infinitepush-write` (kind of like `default-push`, the naming inconsistency is a little unfortunate, as I don't want to use `infinitepush-push`).
The desired behavior of the new path type is as follows:
- takes precedence over `infinitepush` path when the user does `hg push -r . --to scratch/bla`
- replaces `infinitepush` path when the user does `hg push infinitepush -r . --to scratch/bla`
- absence of this path means draft pushes will go to `infinitepush` path
- draft pulls always go to `infinitepush` path, and *there's no fallback to `infinitepush-write`*
- commit cloud always talks to `infinitepush-write`, if it is present (meaning that commit cloud pulls do go to `infinitepush-write` path
- this is done, as commitcloud uses infinitepush paths to also check whether something is backed up
- and also, commitcloud may need to pull very soon after something has been pushed
Reviewed By: quark-zju
Differential Revision: D20368158
fbshipit-source-id: 59db174cebbf2b48765dff37bc93aad176c2d7c1
Summary: [Thrift] Require callers to pass `NetworkSocket` to `TAsyncSocket` instead of relying on `TAsyncSocket` doing the conversion.
Reviewed By: vitaut
Differential Revision: D19800150
fbshipit-source-id: f2b8e76fb16e187810844d8d98654e975cf8dd13
Summary: Those functions are reused in in a future diff.
Reviewed By: sfilipco
Differential Revision: D20367838
fbshipit-source-id: 944babf8c02f0560f8ac8ca5d7c4263432032715
